Do Biased Models Have Biased Thoughts?

• URL: http://arxiv.org/abs/2508.06671v2
• Date: Tue, 12 Aug 2025 02:42:23 GMT
• Title: Do Biased Models Have Biased Thoughts?
• Abstract summary: This paper studies the effect of chain-of-thought prompting, a recent approach that studies the steps followed by the model before it responds, on fairness.<n>To answer our question, we conduct experiments on $5$ popular large language models using fairness metrics to quantify $11$ different biases in the model's thoughts and output.
• Score: 0.48212500317840945
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The impressive performance of language models is undeniable. However, the presence of biases based on gender, race, socio-economic status, physical appearance, and sexual orientation makes the deployment of language models challenging. This paper studies the effect of chain-of-thought prompting, a recent approach that studies the steps followed by the model before it responds, on fairness. More specifically, we ask the following question: $\textit{Do biased models have biased thoughts}$? To answer our question, we conduct experiments on $5$ popular large language models using fairness metrics to quantify $11$ different biases in the model's thoughts and output. Our results show that the bias in the thinking steps is not highly correlated with the output bias (less than $0.6$ correlation with a $p$-value smaller than $0.001$ in most cases). In other words, unlike human beings, the tested models with biased decisions do not always possess biased thoughts.

Related papers

• Towards a more efficient bias detection in financial language models [5.317864735982288]
  Bias in financial language models is a major obstacle to their adoption in real-world applications.<n>We conduct a large-scale study of bias in five financial language models.<n>We explore cross-model-guided bias detection to identify bias-revealing inputs earlier.
  arXiv  Detail & Related papers  (2026-03-09T11:38:53Z)
• Surface Fairness, Deep Bias: A Comparative Study of Bias in Language Models [49.41113560646115]
  We investigate various proxy measures of bias in large language models (LLMs)<n>We find that evaluating models with pre-prompted personae on a multi-subject benchmark (MMLU) leads to negligible and mostly random differences in scores.<n>With the recent trend for LLM assistant memory and personalization, these problems open up from a different angle.
  arXiv  Detail & Related papers  (2025-06-12T08:47:40Z)
• Quantifying Bias in Text-to-Image Generative Models [49.60774626839712]
  Bias in text-to-image (T2I) models can propagate unfair social representations and may be used to aggressively market ideas or push controversial agendas. Existing T2I model bias evaluation methods only focus on social biases. We propose an evaluation methodology to quantify general biases in T2I generative models, without any preconceived notions.
  arXiv  Detail & Related papers  (2023-12-20T14:26:54Z)
• Social Bias Probing: Fairness Benchmarking for Language Models [38.180696489079985]
  This paper proposes a novel framework for probing language models for social biases by assessing disparate treatment. We curate SoFa, a large-scale benchmark designed to address the limitations of existing fairness collections. We show that biases within language models are more nuanced than acknowledged, indicating a broader scope of encoded biases than previously recognized.
  arXiv  Detail & Related papers  (2023-11-15T16:35:59Z)
• Fast Model Debias with Machine Unlearning [54.32026474971696]
  Deep neural networks might behave in a biased manner in many real-world scenarios. Existing debiasing methods suffer from high costs in bias labeling or model re-training. We propose a fast model debiasing framework (FMD) which offers an efficient approach to identify, evaluate and remove biases.
  arXiv  Detail & Related papers  (2023-10-19T08:10:57Z)
• Mitigating Bias for Question Answering Models by Tracking Bias Influence [84.66462028537475]
  We propose BMBI, an approach to mitigate the bias of multiple-choice QA models. Based on the intuition that a model would lean to be more biased if it learns from a biased example, we measure the bias level of a query instance. We show that our method could be applied to multiple QA formulations across multiple bias categories.
  arXiv  Detail & Related papers  (2023-10-13T00:49:09Z)
• OpinionGPT: Modelling Explicit Biases in Instruction-Tuned LLMs [3.5342505775640247]
  We present OpinionGPT, a web demo in which users can ask questions and select all biases they wish to investigate. The demo will answer this question using a model fine-tuned on text representing each of the selected biases. To train the underlying model, we identified 11 different biases (political, geographic, gender, age) and derived an instruction-tuning corpus in which each answer was written by members of one of these demographics.
  arXiv  Detail & Related papers  (2023-09-07T17:41:01Z)
• Echoes: Unsupervised Debiasing via Pseudo-bias Labeling in an Echo Chamber [17.034228910493056]
  This paper presents experimental analyses revealing that the existing biased models overfit to bias-conflicting samples in the training data. We propose a straightforward and effective method called Echoes, which trains a biased model and a target model with a different strategy. Our approach achieves superior debiasing results compared to the existing baselines on both synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2023-05-06T13:13:18Z)
• The SAME score: Improved cosine based bias score for word embeddings [49.75878234192369]
  We introduce SAME, a novel bias score for semantic bias in embeddings. We show that SAME is capable of measuring semantic bias and identify potential causes for social bias in downstream tasks.
  arXiv  Detail & Related papers  (2022-03-28T09:28:13Z)
• Argument from Old Man's View: Assessing Social Bias in Argumentation [20.65183968971417]
  Social bias in language poses a problem with ethical impact for many NLP applications. Recent research has shown that machine learning models trained on respective data may not only adopt, but even amplify the bias. We study the existence of social biases in large English debate portals.
  arXiv  Detail & Related papers  (2020-11-24T10:39:44Z)
• UnQovering Stereotyping Biases via Underspecified Questions [68.81749777034409]
  We present UNQOVER, a framework to probe and quantify biases through underspecified questions. We show that a naive use of model scores can lead to incorrect bias estimates due to two forms of reasoning errors. We use this metric to analyze four important classes of stereotypes: gender, nationality, ethnicity, and religion.
  arXiv  Detail & Related papers  (2020-10-06T01:49:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.