Unexploited Information Value in Human-AI Collaboration

• URL: http://arxiv.org/abs/2411.10463v2
• Date: Tue, 14 Jan 2025 22:08:13 GMT
• Title: Unexploited Information Value in Human-AI Collaboration
• Authors: Ziyang Guo, Yifan Wu, Jason Hartline, Jessica Hullman,
• Abstract summary: How to improve performance of a human-AI team is often not clear without knowing what particular information and strategies each agent employs. We propose a model based in statistical decision theory to analyze human-AI collaboration.
• Score: 23.353778024330165
• License:
• Abstract: Humans and AIs are often paired on decision tasks with the expectation of achieving complementary performance -- where the combination of human and AI outperforms either one alone. However, how to improve performance of a human-AI team is often not clear without knowing more about what particular information and strategies each agent employs. In this paper, we propose a model based in statistical decision theory to analyze human-AI collaboration from the perspective of what information could be used to improve a human or AI decision. We demonstrate our model on a deepfake detection task to investigate seven video-level features by their unexploited value of information. We compare the human alone, AI alone and human-AI team and offer insights on how the AI assistance impacts people's usage of the information and what information that the AI exploits well might be useful for improving human decisions.

Related papers

• The Value of Information in Human-AI Decision-making [23.353778024330165]
  We provide a decision-theoretic framework for characterizing the value of information. We propose a novel information-based instance-level explanation technique.
  arXiv  Detail & Related papers  (2025-02-10T04:50:42Z)
• Utilizing Human Behavior Modeling to Manipulate Explanations in AI-Assisted Decision Making: The Good, the Bad, and the Scary [19.884253335528317]
  Recent advances in AI models have increased the integration of AI-based decision aids into the human decision making process. To fully unlock the potential of AI-assisted decision making, researchers have computationally modeled how humans incorporate AI recommendations into their final decisions. Providing AI explanations to human decision makers to help them rely on AI recommendations more appropriately has become a common practice.
  arXiv  Detail & Related papers  (2024-11-02T18:33:28Z)
• Measuring Human Contribution in AI-Assisted Content Generation [66.06040950325969]
  This study raises the research question of measuring human contribution in AI-assisted content generation. By calculating mutual information between human input and AI-assisted output relative to self-information of AI-assisted output, we quantify the proportional information contribution of humans in content generation.
  arXiv  Detail & Related papers  (2024-08-27T05:56:04Z)
• Towards Human-AI Deliberation: Design and Evaluation of LLM-Empowered Deliberative AI for AI-Assisted Decision-Making [47.33241893184721]
  In AI-assisted decision-making, humans often passively review AI's suggestion and decide whether to accept or reject it as a whole. We propose Human-AI Deliberation, a novel framework to promote human reflection and discussion on conflicting human-AI opinions in decision-making. Based on theories in human deliberation, this framework engages humans and AI in dimension-level opinion elicitation, deliberative discussion, and decision updates.
  arXiv  Detail & Related papers  (2024-03-25T14:34:06Z)
• Human Delegation Behavior in Human-AI Collaboration: The Effect of Contextual Information [7.475784495279183]
  One promising approach to leverage existing complementary capabilities is allowing humans to delegate individual instances of decision tasks to AI. We conduct a behavioral study to explore the effects of providing contextual information to support this delegation decision. Our findings reveal that access to contextual information significantly improves human-AI team performance in delegation settings.
  arXiv  Detail & Related papers  (2024-01-09T18:59:47Z)
• The Impact of Imperfect XAI on Human-AI Decision-Making [8.305869611846775]
  We evaluate how incorrect explanations influence humans' decision-making behavior in a bird species identification task. Our findings reveal the influence of imperfect XAI and humans' level of expertise on their reliance on AI and human-AI team performance.
  arXiv  Detail & Related papers  (2023-07-25T15:19:36Z)
• On the Effect of Information Asymmetry in Human-AI Teams [0.0]
  We focus on the existence of complementarity potential between humans and AI. Specifically, we identify information asymmetry as an essential source of complementarity potential. By conducting an online experiment, we demonstrate that humans can use such contextual information to adjust the AI's decision.
  arXiv  Detail & Related papers  (2022-05-03T13:02:50Z)
• On some Foundational Aspects of Human-Centered Artificial Intelligence [52.03866242565846]
  There is no clear definition of what is meant by Human Centered Artificial Intelligence. This paper introduces the term HCAI agent to refer to any physical or software computational agent equipped with AI components. We see the notion of HCAI agent, together with its components and functions, as a way to bridge the technical and non-technical discussions on human-centered AI.
  arXiv  Detail & Related papers  (2021-12-29T09:58:59Z)
• Trustworthy AI: A Computational Perspective [54.80482955088197]
  We focus on six of the most crucial dimensions in achieving trustworthy AI: (i) Safety & Robustness, (ii) Non-discrimination & Fairness, (iii) Explainability, (iv) Privacy, (v) Accountability & Auditability, and (vi) Environmental Well-Being. For each dimension, we review the recent related technologies according to a taxonomy and summarize their applications in real-world systems.
  arXiv  Detail & Related papers  (2021-07-12T14:21:46Z)
• Is the Most Accurate AI the Best Teammate? Optimizing AI for Teamwork [54.309495231017344]
  We argue that AI systems should be trained in a human-centered manner, directly optimized for team performance. We study this proposal for a specific type of human-AI teaming, where the human overseer chooses to either accept the AI recommendation or solve the task themselves. Our experiments with linear and non-linear models on real-world, high-stakes datasets show that the most accuracy AI may not lead to highest team performance.
  arXiv  Detail & Related papers  (2020-04-27T19:06:28Z)
• Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making [53.62514158534574]
  We study whether features that reveal case-specific model information can calibrate trust and improve the joint performance of the human and AI. We show that confidence score can help calibrate people's trust in an AI model, but trust calibration alone is not sufficient to improve AI-assisted decision making.
  arXiv  Detail & Related papers  (2020-01-07T15:33:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.