Human aversion? Do AI Agents Judge Identity More Harshly Than Performance

• URL: http://arxiv.org/abs/2504.13871v1
• Date: Mon, 31 Mar 2025 02:05:27 GMT
• Title: Human aversion? Do AI Agents Judge Identity More Harshly Than Performance
• Authors: Yuanjun Feng, Vivek Chodhary, Yash Raj Shrestha,
• Abstract summary: We investigate how AI agents based on large language models assess and integrate human input.<n>We find that the AI system systematically discounts human advice, penalizing human errors more severely than algorithmic errors.
• Score: 0.06554326244334868
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This study examines the understudied role of algorithmic evaluation of human judgment in hybrid decision-making systems, a critical gap in management research. While extant literature focuses on human reluctance to follow algorithmic advice, we reverse the perspective by investigating how AI agents based on large language models (LLMs) assess and integrate human input. Our work addresses a pressing managerial constraint: firms barred from deploying LLMs directly due to privacy concerns can still leverage them as mediating tools (for instance, anonymized outputs or decision pipelines) to guide high-stakes choices like pricing or discounts without exposing proprietary data. Through a controlled prediction task, we analyze how an LLM-based AI agent weights human versus algorithmic predictions. We find that the AI system systematically discounts human advice, penalizing human errors more severely than algorithmic errors--a bias exacerbated when the agent's identity (human vs AI) is disclosed and the human is positioned second. These results reveal a disconnect between AI-generated trust metrics and the actual influence of human judgment, challenging assumptions about equitable human-AI collaboration. Our findings offer three key contributions. First, we identify a reverse algorithm aversion phenomenon, where AI agents undervalue human input despite comparable error rates. Second, we demonstrate how disclosure and positional bias interact to amplify this effect, with implications for system design. Third, we provide a framework for indirect LLM deployment that balances predictive power with data privacy. For practitioners, this research emphasize the need to audit AI weighting mechanisms, calibrate trust dynamics, and strategically design decision sequences in human-AI systems.

Related papers

• Human Decision-making is Susceptible to AI-driven Manipulation [87.24007555151452]
  AI systems may exploit users' cognitive biases and emotional vulnerabilities to steer them toward harmful outcomes.<n>This study examined human susceptibility to such manipulation in financial and emotional decision-making contexts.
  arXiv  Detail & Related papers  (2025-02-11T15:56:22Z)
• To Err Is AI! Debugging as an Intervention to Facilitate Appropriate Reliance on AI Systems [11.690126756498223]
  Vision for optimal human-AI collaboration requires 'appropriate reliance' of humans on AI systems. In practice, the performance disparity of machine learning models on out-of-distribution data makes dataset-specific performance feedback unreliable.
  arXiv  Detail & Related papers  (2024-09-22T09:43:27Z)
• Rolling in the deep of cognitive and AI biases [1.556153237434314]
  We argue that there is urgent need to understand AI as a sociotechnical system, inseparable from the conditions in which it is designed, developed and deployed. We address this critical issue by following a radical new methodology under which human cognitive biases become core entities in our AI fairness overview. We introduce a new mapping, which justifies the humans to AI biases and we detect relevant fairness intensities and inter-dependencies.
  arXiv  Detail & Related papers  (2024-07-30T21:34:04Z)
• A Decision Theoretic Framework for Measuring AI Reliance [23.353778024330165]
  Humans frequently make decisions with the aid of artificially intelligent (AI) systems. Researchers have identified ensuring that a human has appropriate reliance on an AI as a critical component of achieving complementary performance. We propose a formal definition of reliance, based on statistical decision theory, which separates the concepts of reliance as the probability the decision-maker follows the AI's recommendation.
  arXiv  Detail & Related papers  (2024-01-27T09:13:09Z)
• Online Decision Mediation [72.80902932543474]
  Consider learning a decision support assistant to serve as an intermediary between (oracle) expert behavior and (imperfect) human behavior. In clinical diagnosis, fully-autonomous machine behavior is often beyond ethical affordances.
  arXiv  Detail & Related papers  (2023-10-28T05:59:43Z)
• Human Uncertainty in Concept-Based AI Systems [37.82747673914624]
  We study human uncertainty in the context of concept-based AI systems. We show that training with uncertain concept labels may help mitigate weaknesses in concept-based systems.
  arXiv  Detail & Related papers  (2023-03-22T19:17:57Z)
• Human-Centric Multimodal Machine Learning: Recent Advances and Testbed on AI-based Recruitment [66.91538273487379]
  There is a certain consensus about the need to develop AI applications with a Human-Centric approach. Human-Centric Machine Learning needs to be developed based on four main requirements: (i) utility and social good; (ii) privacy and data ownership; (iii) transparency and accountability; and (iv) fairness in AI-driven decision-making processes. We study how current multimodal algorithms based on heterogeneous sources of information are affected by sensitive elements and inner biases in the data.
  arXiv  Detail & Related papers  (2023-02-13T16:44:44Z)
• Learning Complementary Policies for Human-AI Teams [22.13683008398939]
  We propose a framework for a novel human-AI collaboration for selecting advantageous course of action. Our solution aims to exploit the human-AI complementarity to maximize decision rewards.
  arXiv  Detail & Related papers  (2023-02-06T17:22:18Z)
• Blessing from Human-AI Interaction: Super Reinforcement Learning in Confounded Environments [19.944163846660498]
  We introduce the paradigm of super reinforcement learning that takes advantage of Human-AI interaction for data driven sequential decision making. In the decision process with unmeasured confounding, the actions taken by past agents can offer valuable insights into undisclosed information. We develop several super-policy learning algorithms and systematically study their theoretical properties.
  arXiv  Detail & Related papers  (2022-09-29T16:03:07Z)
• Bias in Multimodal AI: Testbed for Fair Automatic Recruitment [73.85525896663371]
  We study how current multimodal algorithms based on heterogeneous sources of information are affected by sensitive elements and inner biases in the data. We train automatic recruitment algorithms using a set of multimodal synthetic profiles consciously scored with gender and racial biases. Our methodology and results show how to generate fairer AI-based tools in general, and in particular fairer automated recruitment systems.
  arXiv  Detail & Related papers  (2020-04-15T15:58:05Z)
• A Case for Humans-in-the-Loop: Decisions in the Presence of Erroneous Algorithmic Scores [85.12096045419686]
  We study the adoption of an algorithmic tool used to assist child maltreatment hotline screening decisions. We first show that humans do alter their behavior when the tool is deployed. We show that humans are less likely to adhere to the machine's recommendation when the score displayed is an incorrect estimate of risk.
  arXiv  Detail & Related papers  (2020-02-19T07:27:32Z)
• Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making [53.62514158534574]
  We study whether features that reveal case-specific model information can calibrate trust and improve the joint performance of the human and AI. We show that confidence score can help calibrate people's trust in an AI model, but trust calibration alone is not sufficient to improve AI-assisted decision making.
  arXiv  Detail & Related papers  (2020-01-07T15:33:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.