Addressing Both Statistical and Causal Gender Fairness in NLP Models

• URL: http://arxiv.org/abs/2404.00463v1
• Date: Sat, 30 Mar 2024 20:05:41 GMT
• Title: Addressing Both Statistical and Causal Gender Fairness in NLP Models
• Authors: Hannah Chen, Yangfeng Ji, David Evans,
• Abstract summary: Statistical fairness stipulates equivalent outcomes for every protected group, whereas causal fairness prescribes that a model makes the same prediction for an individual regardless of their protected characteristics. We demonstrate that combinations of statistical and causal debiasing techniques are able to reduce bias measured through both types of metrics.
• Score: 22.75594773147521
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Statistical fairness stipulates equivalent outcomes for every protected group, whereas causal fairness prescribes that a model makes the same prediction for an individual regardless of their protected characteristics. Counterfactual data augmentation (CDA) is effective for reducing bias in NLP models, yet models trained with CDA are often evaluated only on metrics that are closely tied to the causal fairness notion; similarly, sampling-based methods designed to promote statistical fairness are rarely evaluated for causal fairness. In this work, we evaluate both statistical and causal debiasing methods for gender bias in NLP models, and find that while such methods are effective at reducing bias as measured by the targeted metric, they do not necessarily improve results on other bias metrics. We demonstrate that combinations of statistical and causal debiasing techniques are able to reduce bias measured through both types of metrics.

Related papers

• Fast Model Debias with Machine Unlearning [54.32026474971696]
  Deep neural networks might behave in a biased manner in many real-world scenarios. Existing debiasing methods suffer from high costs in bias labeling or model re-training. We propose a fast model debiasing framework (FMD) which offers an efficient approach to identify, evaluate and remove biases.
  arXiv  Detail & Related papers  (2023-10-19T08:10:57Z)
• Balancing Unobserved Confounding with a Few Unbiased Ratings in Debiased Recommendations [4.960902915238239]
  We propose a theoretically guaranteed model-agnostic balancing approach that can be applied to any existing debiasing method. The proposed approach makes full use of unbiased data by alternatively correcting model parameters learned with biased data, and adaptively learning balance coefficients of biased samples for further debiasing.
  arXiv  Detail & Related papers  (2023-04-17T08:56:55Z)
• Counter-GAP: Counterfactual Bias Evaluation through Gendered Ambiguous Pronouns [53.62845317039185]
  Bias-measuring datasets play a critical role in detecting biased behavior of language models. We propose a novel method to collect diverse, natural, and minimally distant text pairs via counterfactual generation. We show that four pre-trained language models are significantly more inconsistent across different gender groups than within each group.
  arXiv  Detail & Related papers  (2023-02-11T12:11:03Z)
• D-BIAS: A Causality-Based Human-in-the-Loop System for Tackling Algorithmic Bias [57.87117733071416]
  We propose D-BIAS, a visual interactive tool that embodies human-in-the-loop AI approach for auditing and mitigating social biases. A user can detect the presence of bias against a group by identifying unfair causal relationships in the causal network. For each interaction, say weakening/deleting a biased causal edge, the system uses a novel method to simulate a new (debiased) dataset.
  arXiv  Detail & Related papers  (2022-08-10T03:41:48Z)
• De-biasing "bias" measurement [20.049916973204102]
  We show that metrics used to measure group-wise model performance disparities are themselves statistically biased estimators of the underlying quantities they purport to represent. We propose the "double-corrected" variance estimator, which provides unbiased estimates and uncertainty quantification of the variance of model performance across groups.
  arXiv  Detail & Related papers  (2022-05-11T20:51:57Z)
• Measuring Fairness of Text Classifiers via Prediction Sensitivity [63.56554964580627]
  ACCUMULATED PREDICTION SENSITIVITY measures fairness in machine learning models based on the model's prediction sensitivity to perturbations in input features. We show that the metric can be theoretically linked with a specific notion of group fairness (statistical parity) and individual fairness.
  arXiv  Detail & Related papers  (2022-03-16T15:00:33Z)
• Information-Theoretic Bias Reduction via Causal View of Spurious Correlation [71.9123886505321]
  We propose an information-theoretic bias measurement technique through a causal interpretation of spurious correlation. We present a novel debiasing framework against the algorithmic bias, which incorporates a bias regularization loss. The proposed bias measurement and debiasing approaches are validated in diverse realistic scenarios.
  arXiv  Detail & Related papers  (2022-01-10T01:19:31Z)
• Information-Theoretic Bias Assessment Of Learned Representations Of Pretrained Face Recognition [18.07966649678408]
  We propose an information-theoretic, independent bias assessment metric to identify degree of bias against protected demographic attributes. Our metric differs from other methods that rely on classification accuracy or examine the differences between ground truth and predicted labels of protected attributes predicted using a shallow network.
  arXiv  Detail & Related papers  (2021-11-08T17:41:17Z)
• Balancing out Bias: Achieving Fairness Through Training Reweighting [58.201275105195485]
  Bias in natural language processing arises from models learning characteristics of the author such as gender and race. Existing methods for mitigating and measuring bias do not directly account for correlations between author demographics and linguistic variables. This paper introduces a very simple but highly effective method for countering bias using instance reweighting.
  arXiv  Detail & Related papers  (2021-09-16T23:40:28Z)
• TARA: Training and Representation Alteration for AI Fairness and Domain Generalization [6.6147550436077776]
  This method uses a dual strategy performing training and representation alteration (TARA) for the mitigation of prominent causes of AI bias. When testing our methods on image analytics, experiments demonstrate that TARA significantly or fully debiases baseline models. recognizing certain limitations in current metrics used for assessing debiasing performance, we propose novel conjunctive debiasing metrics.
  arXiv  Detail & Related papers  (2020-12-11T14:39:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.