AI, Take the Wheel: What Drives Delegation and Trust in Human-Computer Cooperative Question Answering?

• URL: http://arxiv.org/abs/2605.28255v1
• Date: Wed, 27 May 2026 10:05:16 GMT
• Title: AI, Take the Wheel: What Drives Delegation and Trust in Human-Computer Cooperative Question Answering?
• Abstract summary: Humans can make mistakes in deciding whether to trust AI over their own judgment.<n>We study two distinct reliance decisions: the delegation choice and the adoption choice.<n>We find humans make suboptimal collaboration decisions, both under-relying on correct AI suggestions and over-relying when AI misleads them.
• Score: 22.50855320823746
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: AI systems are fallible, and humans can make mistakes in deciding whether to trust AI over their own judgment. Thus, improving human-AI collaboration requires understanding when, why, and how humans decide to rely on AI. We study two distinct reliance decisions: the delegation choice -- deciding when to let AI act autonomously without knowing its output, and the adoption choice -- evaluating AI suggestions and deciding how to use them. Both of these decoupled reliance patterns shape collaboration, but prior work rarely studies them together in realistic settings with the same users. We address this gap by studying collaborative human--AI teams competing in a question-answering game in which humans can choose when and how to work with AI agents to win. Our 24 matches pair 23 expert humans with 16 AI agents, capturing 387 delegation and 1440 adoption decisions. While human--AI collaboration performs better than either AI or humans alone, humans make suboptimal collaboration decisions, both under-relying on correct AI suggestions (3.9% of opportunities missed) and over-relying when AI misleads them (1.7%). Both parties contribute wrong answers: reported model confidence is near chance when humans and AI disagree, while confirmation bias drives higher under-reliance (64.5%) when an AI suggestion agrees with humans' initial incorrect answer. To close this gap, we recommend calibrated confidence, evidence-grounded explanations, and mechanisms that help users refine trust.

Related papers

• Toward Human-AI Complementarity Across Diverse Tasks [3.6671496239276475]
  Human-AI complementarity is the idea that combining human and AI judgments can outperform either alone.<n>We investigate whether human-AI complementarity can be achieved on realistic tasks.<n>We find only modest complementarity gains.
  arXiv  Detail & Related papers  (2026-04-13T07:24:18Z)
• How Performance Pressure Influences AI-Assisted Decision Making [52.997197698288936]
  We show how pressure and explainable AI (XAI) techniques interact with AI advice-taking behavior.<n>Our results show complex interaction effects, with different combinations of pressure and XAI techniques either improving or worsening AI advice taking behavior.
  arXiv  Detail & Related papers  (2024-10-21T22:39:52Z)
• Improving Human-AI Collaboration With Descriptions of AI Behavior [14.904401331154062]
  People work with AI systems to improve their decision making, but often under- or over-rely on AI predictions and perform worse than they would have unassisted. To help people appropriately rely on AI aids, we propose showing them behavior descriptions.
  arXiv  Detail & Related papers  (2023-01-06T00:33:08Z)
• Measuring an artificial intelligence agent's trust in humans using machine incentives [2.1016374925364616]
  Gauging an AI agent's trust in humans is challenging because dishonesty might respond falsely about their trust in humans. We present a method for incentivizing machine decisions without altering an AI agent's underlying algorithms or goal orientation. Our experiments suggest that one of the most advanced AI language models to date alters its social behavior in response to incentives.
  arXiv  Detail & Related papers  (2022-12-27T06:05:49Z)
• Can Machines Imitate Humans? Integrative Turing-like tests for Language and Vision Demonstrate a Narrowing Gap [56.611702960809644]
  We benchmark AI's ability to imitate humans in three language tasks and three vision tasks.<n>Next, we conducted 72,191 Turing-like tests with 1,916 human judges and 10 AI judges.<n>Imitation ability showed minimal correlation with conventional AI performance metrics.
  arXiv  Detail & Related papers  (2022-11-23T16:16:52Z)
• Uncalibrated Models Can Improve Human-AI Collaboration [10.106324182884068]
  We show that presenting AI models as more confident than they actually are can improve human-AI performance. We first learn a model for how humans incorporate AI advice using data from thousands of human interactions.
  arXiv  Detail & Related papers  (2022-02-12T04:51:00Z)
• Cybertrust: From Explainable to Actionable and Interpretable AI (AI2) [58.981120701284816]
  Actionable and Interpretable AI (AI2) will incorporate explicit quantifications and visualizations of user confidence in AI recommendations. It will allow examining and testing of AI system predictions to establish a basis for trust in the systems' decision making.
  arXiv  Detail & Related papers  (2022-01-26T18:53:09Z)
• Trustworthy AI: A Computational Perspective [54.80482955088197]
  We focus on six of the most crucial dimensions in achieving trustworthy AI: (i) Safety & Robustness, (ii) Non-discrimination & Fairness, (iii) Explainability, (iv) Privacy, (v) Accountability & Auditability, and (vi) Environmental Well-Being. For each dimension, we review the recent related technologies according to a taxonomy and summarize their applications in real-world systems.
  arXiv  Detail & Related papers  (2021-07-12T14:21:46Z)
• Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance [44.730580857733]
  Prior studies observed improvements from explanations only when the AI, alone, outperformed both the human and the best team. We conduct mixed-method user studies on three datasets, where an AI with accuracy comparable to humans helps participants solve a task. We find explanations increase the chance that humans will accept the AI's recommendation, regardless of its correctness.
  arXiv  Detail & Related papers  (2020-06-26T03:34:04Z)
• Is the Most Accurate AI the Best Teammate? Optimizing AI for Teamwork [54.309495231017344]
  We argue that AI systems should be trained in a human-centered manner, directly optimized for team performance. We study this proposal for a specific type of human-AI teaming, where the human overseer chooses to either accept the AI recommendation or solve the task themselves. Our experiments with linear and non-linear models on real-world, high-stakes datasets show that the most accuracy AI may not lead to highest team performance.
  arXiv  Detail & Related papers  (2020-04-27T19:06:28Z)
• Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making [53.62514158534574]
  We study whether features that reveal case-specific model information can calibrate trust and improve the joint performance of the human and AI. We show that confidence score can help calibrate people's trust in an AI model, but trust calibration alone is not sufficient to improve AI-assisted decision making.
  arXiv  Detail & Related papers  (2020-01-07T15:33:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.