Mechanics of Bias and Reasoning: Interpreting the Impact of Chain-of-Thought Prompting on Gender Bias in LLMs

• URL: http://arxiv.org/abs/2605.20410v1
• Date: Tue, 19 May 2026 19:05:13 GMT
• Title: Mechanics of Bias and Reasoning: Interpreting the Impact of Chain-of-Thought Prompting on Gender Bias in LLMs
• Abstract summary: Chain-of-Thought (CoT) prompting has been proposed as a bias-mitigation approach.<n>Mechanistic analyses reveal that although CoT balances biased behavior in certain attention head clusters, gender bias remains embedded in hidden representations.
• Score: 9.546109127678625
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) are increasingly deployed in socially sensitive settings despite substantial documentation that they encode gender biases. Chain-of-Thought (CoT) prompting has been proposed as a bias-mitigation approach. However, existing evaluations primarily focus on changes in LLM benchmark performance, providing limited insight into whether apparent bias reductions reflect meaningful changes in a model's internal mechanisms. In this work, we investigate how CoT prompting affects gender bias in LLMs, combining benchmark-based evaluation with mechanistic interpretability techniques and reasoning chain failure analysis. Our results confirm a stereotypical bias present in LLM outputs across benchmarks, showing that CoT prompting does not consistently reduce the bias gap. Mechanistic analyses reveal that although CoT balances biased behavior in certain attention head clusters, gender bias remains embedded in hidden representations, indicating only superficial mitigation. Inspection of reasoning chains further suggests that these improvements stem from memorization and familiarity with the dataset rather than genuine understanding of bias.

Related papers

• Alignment Reduces Expressed but Not Encoded Gender Bias: A Unified Framework and Study [3.679036235271287]
  We propose a unified framework to jointly analyze intrinsic and extrinsic gender bias in Large Language Models.<n>We find a consistent association between latent gender information and expressed bias when measured under the unified protocol.<n>Our results suggest that while the latter indeed reduces expressed bias, measurable gender-related associations are still present in internal representations.
  arXiv  Detail & Related papers  (2026-03-25T09:35:18Z)
• Evaluating Social Bias in RAG Systems: When External Context Helps and Reasoning Hurts [7.344577590113121]
  Social biases inherent in large language models (LLMs) raise significant fairness concerns.<n>This work focuses on evaluating and understanding the social bias implications of RAG.
  arXiv  Detail & Related papers  (2026-02-10T06:27:56Z)
• Bias in Gender Bias Benchmarks: How Spurious Features Distort Evaluation [116.86965910589775]
  We show that even minimal perturbations, such as masking just 10% of objects or weakly blurring backgrounds, can dramatically alter bias scores.<n>This suggests that current bias evaluations reflect model responses to spurious features rather than gender bias.
  arXiv  Detail & Related papers  (2025-09-09T11:14:11Z)
• Measuring Bias or Measuring the Task: Understanding the Brittle Nature of LLM Gender Biases [2.9803250365852443]
  This paper examines how signaling the evaluative purpose of a task impacts measured gender bias in LLMs.<n>We find that prompts that more clearly align with (gender bias) evaluation framing elicit distinct gender output distributions.
  arXiv  Detail & Related papers  (2025-09-04T16:32:18Z)
• A Closer Look at Bias and Chain-of-Thought Faithfulness of Large (Vision) Language Models [58.32070787537946]
  Chain-of-thought (CoT) reasoning enhances performance of large language models.<n>We present the first comprehensive study of CoT faithfulness in large vision-language models.
  arXiv  Detail & Related papers  (2025-05-29T18:55:05Z)
• Does Reasoning Introduce Bias? A Study of Social Bias Evaluation and Mitigation in LLM Reasoning [21.921684911834447]
  We present the first systematic evaluation of social bias within large language models (LLMs)<n>We analyze both prediction accuracy and reasoning bias across a broad spectrum of models, including instruction-tuned and CoT-augmented variants of DeepSeek-R1 and ChatGPT.<n>We propose Answer Distribution as Bias Proxy (ADBP), a lightweight mitigation method that detects bias by tracking how model predictions change across incremental reasoning steps.
  arXiv  Detail & Related papers  (2025-02-21T10:16:07Z)
• How far can bias go? -- Tracing bias from pretraining data to alignment [54.51310112013655]
  This study examines the correlation between gender-occupation bias in pre-training data and their manifestation in LLMs.<n>Our findings reveal that biases present in pre-training data are amplified in model outputs.
  arXiv  Detail & Related papers  (2024-11-28T16:20:25Z)
• The Root Shapes the Fruit: On the Persistence of Gender-Exclusive Harms in Aligned Language Models [91.86718720024825]
  We center transgender, nonbinary, and other gender-diverse identities to investigate how alignment procedures interact with pre-existing gender-diverse bias.<n>Our findings reveal that DPO-aligned models are particularly sensitive to supervised finetuning.<n>We conclude with recommendations tailored to DPO and broader alignment practices.
  arXiv  Detail & Related papers  (2024-11-06T06:50:50Z)
• GenderCARE: A Comprehensive Framework for Assessing and Reducing Gender Bias in Large Language Models [73.23743278545321]
  Large language models (LLMs) have exhibited remarkable capabilities in natural language generation, but have also been observed to magnify societal biases.<n>GenderCARE is a comprehensive framework that encompasses innovative Criteria, bias Assessment, Reduction techniques, and Evaluation metrics.
  arXiv  Detail & Related papers  (2024-08-22T15:35:46Z)
• Identifying and Mitigating Social Bias Knowledge in Language Models [52.52955281662332]
  We propose a novel debiasing approach, Fairness Stamp (FAST), which enables fine-grained calibration of individual social biases.<n>FAST surpasses state-of-the-art baselines with superior debiasing performance.<n>This highlights the potential of fine-grained debiasing strategies to achieve fairness in large language models.
  arXiv  Detail & Related papers  (2024-08-07T17:14:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.