No Thoughts Just AI: Biased LLM Hiring Recommendations Alter Human Decision Making and Limit Human Autonomy

• URL: http://arxiv.org/abs/2509.04404v2
• Date: Mon, 08 Sep 2025 19:40:40 GMT
• Title: No Thoughts Just AI: Biased LLM Hiring Recommendations Alter Human Decision Making and Limit Human Autonomy
• Authors: Kyra Wilson, Mattea Sim, Anna-Maria Gueorguieva, Aylin Caliskan,
• Abstract summary: People collaborate with simulated AI models exhibiting race-based preferences (bias) to evaluate candidates for 16 high and low status occupations.<n>Simulated AI bias approximates factual and counterfactual estimates of racial bias in real-world AI systems.
• Score: 8.423021413553464
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: In this study, we conduct a resume-screening experiment (N=528) where people collaborate with simulated AI models exhibiting race-based preferences (bias) to evaluate candidates for 16 high and low status occupations. Simulated AI bias approximates factual and counterfactual estimates of racial bias in real-world AI systems. We investigate people's preferences for White, Black, Hispanic, and Asian candidates (represented through names and affinity groups on quality-controlled resumes) across 1,526 scenarios and measure their unconscious associations between race and status using implicit association tests (IATs), which predict discriminatory hiring decisions but have not been investigated in human-AI collaboration. When making decisions without AI or with AI that exhibits no race-based preferences, people select all candidates at equal rates. However, when interacting with AI favoring a particular group, people also favor those candidates up to 90% of the time, indicating a significant behavioral shift. The likelihood of selecting candidates whose identities do not align with common race-status stereotypes can increase by 13% if people complete an IAT before conducting resume screening. Finally, even if people think AI recommendations are low quality or not important, their decisions are still vulnerable to AI bias under certain circumstances. This work has implications for people's autonomy in AI-HITL scenarios, AI and work, design and evaluation of AI hiring systems, and strategies for mitigating bias in collaborative decision-making tasks. In particular, organizational and regulatory policy should acknowledge the complex nature of AI-HITL decision making when implementing these systems, educating people who use them, and determining which are subject to oversight.

Related papers

• Measuring Political Preferences in AI Systems: An Integrative Approach [0.0]
  This study employs a multi-method approach to assess political bias in leading AI systems.<n>Results indicate a consistent left-leaning bias across most contemporary AI systems.<n>The presence of systematic political bias in AI systems poses risks, including reduced viewpoint diversity, increased societal polarization, and the potential for public mistrust in AI technologies.
  arXiv  Detail & Related papers  (2025-03-04T01:40:28Z)
• Nationality, Race, and Ethnicity Biases in and Consequences of Detecting AI-Generated Self-Presentations [1.6772190302364975]
  Content, such as linguistic style, played a dominant role in AI detection.<n>Asian and Hispanic applicants were more likely to be judged as AI users when labeled as domestic students.
  arXiv  Detail & Related papers  (2024-12-24T18:31:44Z)
• Biased AI can Influence Political Decision-Making [64.9461133083473]
  This paper presents two experiments investigating the effects of partisan bias in large language models (LLMs) on political opinions and decision-making.<n>We found that participants exposed to partisan biased models were significantly more likely to adopt opinions and make decisions which matched the LLM's bias.
  arXiv  Detail & Related papers  (2024-10-08T22:56:00Z)
• Particip-AI: A Democratic Surveying Framework for Anticipating Future AI Use Cases, Harms and Benefits [54.648819983899614]
  General purpose AI seems to have lowered the barriers for the public to use AI and harness its power. We introduce PARTICIP-AI, a framework for laypeople to speculate and assess AI use cases and their impacts.
  arXiv  Detail & Related papers  (2024-03-21T19:12:37Z)
• Training Towards Critical Use: Learning to Situate AI Predictions Relative to Human Knowledge [22.21959942886099]
  We introduce a process-oriented notion of appropriate reliance called critical use that centers the human's ability to situate AI predictions against knowledge that is uniquely available to them but unavailable to the AI model. We conduct a randomized online experiment in a complex social decision-making setting: child maltreatment screening. We find that, by providing participants with accelerated, low-stakes opportunities to practice AI-assisted decision-making, novices came to exhibit patterns of disagreement with AI that resemble those of experienced workers.
  arXiv  Detail & Related papers  (2023-08-30T01:54:31Z)
• Bending the Automation Bias Curve: A Study of Human and AI-based Decision Making in National Security Contexts [0.0]
  We theorize about the relationship between background knowledge about AI, trust in AI, and how these interact with other factors to influence the probability of automation bias. We test these in a preregistered task identification experiment across a representative sample of 9000 adults in 9 countries with varying levels of AI industries.
  arXiv  Detail & Related papers  (2023-06-28T18:57:36Z)
• Fairness in AI and Its Long-Term Implications on Society [68.8204255655161]
  We take a closer look at AI fairness and analyze how lack of AI fairness can lead to deepening of biases over time. We discuss how biased models can lead to more negative real-world outcomes for certain groups. If the issues persist, they could be reinforced by interactions with other risks and have severe implications on society in the form of social unrest.
  arXiv  Detail & Related papers  (2023-04-16T11:22:59Z)
• Human-Centric Multimodal Machine Learning: Recent Advances and Testbed on AI-based Recruitment [66.91538273487379]
  There is a certain consensus about the need to develop AI applications with a Human-Centric approach. Human-Centric Machine Learning needs to be developed based on four main requirements: (i) utility and social good; (ii) privacy and data ownership; (iii) transparency and accountability; and (iv) fairness in AI-driven decision-making processes. We study how current multimodal algorithms based on heterogeneous sources of information are affected by sensitive elements and inner biases in the data.
  arXiv  Detail & Related papers  (2023-02-13T16:44:44Z)
• How Different Groups Prioritize Ethical Values for Responsible AI [75.40051547428592]
  Private companies, public sector organizations, and academic groups have outlined ethical values they consider important for responsible AI technologies. While their recommendations converge on a set of central values, little is known about the values a more representative public would find important for the AI technologies they interact with and might be affected by. We conducted a survey examining how individuals perceive and prioritize responsible AI values across three groups.
  arXiv  Detail & Related papers  (2022-05-16T14:39:37Z)
• Bias and Discrimination in AI: a cross-disciplinary perspective [5.190307793476366]
  We show that finding solutions to bias and discrimination in AI requires robust cross-disciplinary collaborations. We survey relevant literature about bias and discrimination in AI from an interdisciplinary perspective that embeds technical, legal, social and ethical dimensions.
  arXiv  Detail & Related papers  (2020-08-11T10:02:04Z)
• Is the Most Accurate AI the Best Teammate? Optimizing AI for Teamwork [54.309495231017344]
  We argue that AI systems should be trained in a human-centered manner, directly optimized for team performance. We study this proposal for a specific type of human-AI teaming, where the human overseer chooses to either accept the AI recommendation or solve the task themselves. Our experiments with linear and non-linear models on real-world, high-stakes datasets show that the most accuracy AI may not lead to highest team performance.
  arXiv  Detail & Related papers  (2020-04-27T19:06:28Z)
• Bias in Multimodal AI: Testbed for Fair Automatic Recruitment [73.85525896663371]
  We study how current multimodal algorithms based on heterogeneous sources of information are affected by sensitive elements and inner biases in the data. We train automatic recruitment algorithms using a set of multimodal synthetic profiles consciously scored with gender and racial biases. Our methodology and results show how to generate fairer AI-based tools in general, and in particular fairer automated recruitment systems.
  arXiv  Detail & Related papers  (2020-04-15T15:58:05Z)
• Bias in Data-driven AI Systems -- An Introductory Survey [37.34717604783343]
  This survey focuses on data-driven AI, as a large part of AI is powered nowadays by (big) data and powerful Machine Learning (ML) algorithms. If otherwise not specified, we use the general term bias to describe problems related to the gathering or processing of data that might result in prejudiced decisions on the bases of demographic features like race, sex, etc.
  arXiv  Detail & Related papers  (2020-01-14T09:39:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.