The Impact of Unstated Norms in Bias Analysis of Language Models

• URL: http://arxiv.org/abs/2404.03471v3
• Date: Fri, 27 Sep 2024 13:12:23 GMT
• Title: The Impact of Unstated Norms in Bias Analysis of Language Models
• Authors: Farnaz Kohankhaki, D. B. Emerson, Jacob-Junqi Tian, Laleh Seyyed-Kalantari, Faiza Khan Khattak,
• Abstract summary: Counterfactual bias evaluation is a widely used approach to quantifying bias. We find that template-based probes can lead to unrealistic bias measurements.
• Score: 0.03495246564946556
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Bias in large language models (LLMs) has many forms, from overt discrimination to implicit stereotypes. Counterfactual bias evaluation is a widely used approach to quantifying bias and often relies on template-based probes that explicitly state group membership. It measures whether the outcome of a task, performed by an LLM, is invariant to a change of group membership. In this work, we find that template-based probes can lead to unrealistic bias measurements. For example, LLMs appear to mistakenly cast text associated with White race as negative at higher rates than other groups. We hypothesize that this arises artificially via a mismatch between commonly unstated norms, in the form of markedness, in the pretraining text of LLMs (e.g., Black president vs. president) and templates used for bias measurement (e.g., Black president vs. White president). The findings highlight the potential misleading impact of varying group membership through explicit mention in counterfactual bias quantification.

Related papers

• Explicit vs. Implicit: Investigating Social Bias in Large Language Models through Self-Reflection [5.800102484016876]
  Large Language Models (LLMs) have been shown to exhibit various biases and stereotypes in their generated content. This paper presents a systematic framework grounded in social psychology theories to investigate explicit and implicit biases in LLMs.
  arXiv  Detail & Related papers  (2025-01-04T14:08:52Z)
• How far can bias go? -- Tracing bias from pretraining data to alignment [54.51310112013655]
  This study examines the correlation between gender-occupation bias in pre-training data and their manifestation in LLMs. Our findings reveal that biases present in pre-training data are amplified in model outputs.
  arXiv  Detail & Related papers  (2024-11-28T16:20:25Z)
• A Novel Interpretability Metric for Explaining Bias in Language Models: Applications on Multilingual Models from Southeast Asia [0.3376269351435396]
  We propose a novel metric to measure token-level contributions to biased behavior in pretrained language models (PLMs) Our results confirm the presence of sexist and homophobic bias in Southeast Asian PLMs. Interpretability and semantic analyses also reveal that PLM bias is strongly induced by words relating to crime, intimate relationships, and helping.
  arXiv  Detail & Related papers  (2024-10-20T18:31:05Z)
• Promoting Equality in Large Language Models: Identifying and Mitigating the Implicit Bias based on Bayesian Theory [29.201402717025335]
  Large language models (LLMs) are trained on extensive text corpora, which inevitably include biased information. We have formally defined the implicit bias problem and developed an innovative framework for bias removal based on Bayesian theory.
  arXiv  Detail & Related papers  (2024-08-20T07:40:12Z)
• White Men Lead, Black Women Help? Benchmarking Language Agency Social Biases in LLMs [58.27353205269664]
  Social biases can manifest in language agency. We introduce the novel Language Agency Bias Evaluation benchmark. We unveil language agency social biases in 3 recent Large Language Model (LLM)-generated content.
  arXiv  Detail & Related papers  (2024-04-16T12:27:54Z)
• Bias in Language Models: Beyond Trick Tests and Toward RUTEd Evaluation [49.3814117521631]
  Standard benchmarks of bias and fairness in large language models (LLMs) measure the association between social attributes implied in user prompts and short responses. We develop analogous RUTEd evaluations from three contexts of real-world use. We find that standard bias metrics have no significant correlation with the more realistic bias metrics.
  arXiv  Detail & Related papers  (2024-02-20T01:49:15Z)
• GPTBIAS: A Comprehensive Framework for Evaluating Bias in Large Language Models [83.30078426829627]
  Large language models (LLMs) have gained popularity and are being widely adopted by a large user community. The existing evaluation methods have many constraints, and their results exhibit a limited degree of interpretability. We propose a bias evaluation framework named GPTBIAS that leverages the high performance of LLMs to assess bias in models.
  arXiv  Detail & Related papers  (2023-12-11T12:02:14Z)
• Shedding light on underrepresentation and Sampling Bias in machine learning [0.0]
  We show how discrimination can be decomposed into variance, bias, and noise. We challenge the commonly accepted mitigation approach that discrimination can be addressed by collecting more samples of the underrepresented group.
  arXiv  Detail & Related papers  (2023-06-08T09:34:20Z)
• Balancing out Bias: Achieving Fairness Through Training Reweighting [58.201275105195485]
  Bias in natural language processing arises from models learning characteristics of the author such as gender and race. Existing methods for mitigating and measuring bias do not directly account for correlations between author demographics and linguistic variables. This paper introduces a very simple but highly effective method for countering bias using instance reweighting.
  arXiv  Detail & Related papers  (2021-09-16T23:40:28Z)
• LOGAN: Local Group Bias Detection by Clustering [86.38331353310114]
  We argue that evaluating bias at the corpus level is not enough for understanding how biases are embedded in a model. We propose LOGAN, a new bias detection technique based on clustering. Experiments on toxicity classification and object classification tasks show that LOGAN identifies bias in a local region.
  arXiv  Detail & Related papers  (2020-10-06T16:42:51Z)
• Detecting Emergent Intersectional Biases: Contextualized Word Embeddings Contain a Distribution of Human-like Biases [10.713568409205077]
  State-of-the-art neural language models generate dynamic word embeddings dependent on the context in which the word appears. We introduce the Contextualized Embedding Association Test (CEAT), that can summarize the magnitude of overall bias in neural language models. We develop two methods, Intersectional Bias Detection (IBD) and Emergent Intersectional Bias Detection (EIBD), to automatically identify the intersectional biases and emergent intersectional biases from static word embeddings.
  arXiv  Detail & Related papers  (2020-06-06T19:49:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.