When Bias Backfires: The Modulatory Role of Counterfactual Explanations on the Adoption of Algorithmic Bias in XAI-Supported Human Decision-Making

• URL: http://arxiv.org/abs/2505.14377v1
• Date: Tue, 20 May 2025 14:00:28 GMT
• Title: When Bias Backfires: The Modulatory Role of Counterfactual Explanations on the Adoption of Algorithmic Bias in XAI-Supported Human Decision-Making
• Authors: Ulrike Kuhl, Annika Bush,
• Abstract summary: This study simulated hiring decisions to examine how biased AI recommendations influence human judgment over time.<n>Our results indicate that the participants followed the AI recommendations 70% of the time when the qualifications of the given candidates were comparable.<n>In the post-interaction phase, participants' independent decisions aligned with the bias when no counterfactual explanations were provided before, but reversed the bias when explanations were given.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Although the integration of artificial intelligence (AI) into everyday tasks improves efficiency and objectivity, it also risks transmitting bias to human decision-making. In this study, we conducted a controlled experiment that simulated hiring decisions to examine how biased AI recommendations - augmented with or without counterfactual explanations - influence human judgment over time. Participants, acting as hiring managers, completed 60 decision trials divided into a baseline phase without AI, followed by a phase with biased (X)AI recommendations (favoring either male or female candidates), and a final post-interaction phase without AI. Our results indicate that the participants followed the AI recommendations 70% of the time when the qualifications of the given candidates were comparable. Yet, only a fraction of participants detected the gender bias (8 out of 294). Crucially, exposure to biased AI altered participants' inherent preferences: in the post-interaction phase, participants' independent decisions aligned with the bias when no counterfactual explanations were provided before, but reversed the bias when explanations were given. Reported trust did not differ significantly across conditions. Confidence varied throughout the study phases after exposure to male-biased AI, indicating nuanced effects of AI bias on decision certainty. Our findings point to the importance of calibrating XAI to avoid unintended behavioral shifts in order to safeguard equitable decision-making and prevent the adoption of algorithmic bias.

Related papers

• Human Decision-making is Susceptible to AI-driven Manipulation [87.24007555151452]
  AI systems may exploit users' cognitive biases and emotional vulnerabilities to steer them toward harmful outcomes.<n>This study examined human susceptibility to such manipulation in financial and emotional decision-making contexts.
  arXiv  Detail & Related papers  (2025-02-11T15:56:22Z)
• Human-Alignment Influences the Utility of AI-assisted Decision Making [16.732483972136418]
  We investigate what extent the degree of alignment actually influences the utility of AI-assisted decision making.<n>Our results show a positive association between the degree of alignment and the utility of AI-assisted decision making.
  arXiv  Detail & Related papers  (2025-01-23T19:01:47Z)
• Using Machine Bias To Measure Human Bias [10.983607899068204]
  We propose a machine learning-based framework to assess bias in human-generated decisions.<n>We provide theoretical guarantees and empirical evidence demonstrating the superiority of our method over existing alternatives.
  arXiv  Detail & Related papers  (2024-11-27T08:02:31Z)
• Biased AI can Influence Political Decision-Making [64.9461133083473]
  This paper presents two experiments investigating the effects of partisan bias in AI language models on political decision-making. We found that participants exposed to politically biased models were significantly more likely to adopt opinions and make decisions aligning with the AI's bias.
  arXiv  Detail & Related papers  (2024-10-08T22:56:00Z)
• Overcoming Anchoring Bias: The Potential of AI and XAI-based Decision Support [0.0]
  Information systems (IS) are frequently designed to leverage the negative effect of anchoring bias to influence individuals' decision-making. Recent advances in Artificial Intelligence (AI) have opened new opportunities for mitigating biased decisions.
  arXiv  Detail & Related papers  (2024-05-08T11:25:04Z)
• Underspecified Human Decision Experiments Considered Harmful [18.27590643693167]
  We present a widely applicable definition of a decision problem synthesized from statistical decision theory and information economics.<n>We evaluate whether recent empirical research on AI-assisted decisions achieves this standard.
  arXiv  Detail & Related papers  (2024-01-25T16:21:37Z)
• Online Decision Mediation [72.80902932543474]
  Consider learning a decision support assistant to serve as an intermediary between (oracle) expert behavior and (imperfect) human behavior. In clinical diagnosis, fully-autonomous machine behavior is often beyond ethical affordances.
  arXiv  Detail & Related papers  (2023-10-28T05:59:43Z)
• Decision-makers Processing of AI Algorithmic Advice: Automation Bias versus Selective Adherence [0.0]
  Key concern is that human overreliance on algorithms introduces new biases in the human-algorithm interaction. A second concern regards decision-makers inclination to selectively adopt algorithmic advice when it matches their pre-existing beliefs and stereotypes. We assess these via two studies simulating the use of algorithmic advice in decisions pertaining to the employment of school teachers in the Netherlands. Our findings of selective, biased adherence belie the promise of neutrality that has propelled algorithm use in the public sector.
  arXiv  Detail & Related papers  (2021-03-03T13:10:50Z)
• Deciding Fast and Slow: The Role of Cognitive Biases in AI-assisted Decision-making [46.625616262738404]
  We use knowledge from the field of cognitive science to account for cognitive biases in the human-AI collaborative decision-making setting. We focus specifically on anchoring bias, a bias commonly encountered in human-AI collaboration.
  arXiv  Detail & Related papers  (2020-10-15T22:25:41Z)
• Towards causal benchmarking of bias in face analysis algorithms [54.19499274513654]
  We develop an experimental method for measuring algorithmic bias of face analysis algorithms. Our proposed method is based on generating synthetic transects'' of matched sample images. We validate our method by comparing it to a study that employs the traditional observational method for analyzing bias in gender classification algorithms.
  arXiv  Detail & Related papers  (2020-07-13T17:10:34Z)
• Bias in Multimodal AI: Testbed for Fair Automatic Recruitment [73.85525896663371]
  We study how current multimodal algorithms based on heterogeneous sources of information are affected by sensitive elements and inner biases in the data. We train automatic recruitment algorithms using a set of multimodal synthetic profiles consciously scored with gender and racial biases. Our methodology and results show how to generate fairer AI-based tools in general, and in particular fairer automated recruitment systems.
  arXiv  Detail & Related papers  (2020-04-15T15:58:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.