A Bayesian approach to uncertainty in word embedding bias estimation

• URL: http://arxiv.org/abs/2306.09066v1
• Date: Thu, 15 Jun 2023 11:48:50 GMT
• Title: A Bayesian approach to uncertainty in word embedding bias estimation
• Authors: Alicja Dobrzeniecka and Rafal Urbaniak
• Abstract summary: Multiple measures, such as WEAT or MAC, attempt to quantify the magnitude of bias present in word embeddings in terms of a single-number metric. We show that similar results can be easily obtained using such methods even if the data are generated by a null model lacking the intended bias. We propose a Bayesian alternative: hierarchical Bayesian modeling, which enables a more uncertainty-sensitive inspection of bias in word embeddings.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Multiple measures, such as WEAT or MAC, attempt to quantify the magnitude of bias present in word embeddings in terms of a single-number metric. However, such metrics and the related statistical significance calculations rely on treating pre-averaged data as individual data points and employing bootstrapping techniques with low sample sizes. We show that similar results can be easily obtained using such methods even if the data are generated by a null model lacking the intended bias. Consequently, we argue that this approach generates false confidence. To address this issue, we propose a Bayesian alternative: hierarchical Bayesian modeling, which enables a more uncertainty-sensitive inspection of bias in word embeddings at different levels of granularity. To showcase our method, we apply it to Religion, Gender, and Race word lists from the original research, together with our control neutral word lists. We deploy the method using Google, Glove, and Reddit embeddings. Further, we utilize our approach to evaluate a debiasing technique applied to Reddit word embedding. Our findings reveal a more complex landscape than suggested by the proponents of single-number metrics. The datasets and source code for the paper are publicly available.

Related papers

• Statistical Uncertainty in Word Embeddings: GloVe-V [35.04183792123882]
  We introduce a method to obtain approximate, easy-to-use, and scalable reconstruction error variance estimates for GloVe. To demonstrate the value of embeddings with variance (GloVe-V), we illustrate how our approach enables principled hypothesis testing in core word embedding tasks.
  arXiv  Detail & Related papers  (2024-06-18T00:35:02Z)
• Debiasing Sentence Embedders through Contrastive Word Pairs [46.9044612783003]
  We explore an approach to remove linear and nonlinear bias information for NLP solutions. We compare our approach to common debiasing methods on classical bias metrics and on bias metrics which take nonlinear information into account.
  arXiv  Detail & Related papers  (2024-03-27T13:34:59Z)
• Revisiting the Dataset Bias Problem from a Statistical Perspective [72.94990819287551]
  We study the "dataset bias" problem from a statistical standpoint. We identify the main cause of the problem as the strong correlation between a class attribute u and a non-class attribute b. We propose to mitigate dataset bias via either weighting the objective of each sample n by frac1p(u_n|b_n) or sampling that sample with a weight proportional to frac1p(u_n|b_n).
  arXiv  Detail & Related papers  (2024-02-05T22:58:06Z)
• Counter-GAP: Counterfactual Bias Evaluation through Gendered Ambiguous Pronouns [53.62845317039185]
  Bias-measuring datasets play a critical role in detecting biased behavior of language models. We propose a novel method to collect diverse, natural, and minimally distant text pairs via counterfactual generation. We show that four pre-trained language models are significantly more inconsistent across different gender groups than within each group.
  arXiv  Detail & Related papers  (2023-02-11T12:11:03Z)
• MABEL: Attenuating Gender Bias using Textual Entailment Data [20.489427903240017]
  We propose MABEL, an intermediate pre-training approach for mitigating gender bias in contextualized representations. Key to our approach is the use of a contrastive learning objective on counterfactually augmented, gender-balanced entailment pairs. We show that MABEL outperforms previous task-agnostic debiasing approaches in terms of fairness.
  arXiv  Detail & Related papers  (2022-10-26T18:36:58Z)
• Evaluating Metrics for Bias in Word Embeddings [44.14639209617701]
  We formalize a bias definition based on the ideas from previous works and derive conditions for bias metrics. We propose a new metric, SAME, to address the shortcomings of existing metrics and mathematically prove that SAME behaves appropriately.
  arXiv  Detail & Related papers  (2021-11-15T16:07:15Z)
• The Gap on GAP: Tackling the Problem of Differing Data Distributions in Bias-Measuring Datasets [58.53269361115974]
  Diagnostic datasets that can detect biased models are an important prerequisite for bias reduction within natural language processing. undesired patterns in the collected data can make such tests incorrect. We introduce a theoretically grounded method for weighting test samples to cope with such patterns in the test data.
  arXiv  Detail & Related papers  (2020-11-03T16:50:13Z)
• Double-Hard Debias: Tailoring Word Embeddings for Gender Bias Mitigation [94.98656228690233]
  We propose a technique that purifies the word embeddings against corpus regularities prior to inferring and removing the gender subspace. Our approach preserves the distributional semantics of the pre-trained word embeddings while reducing gender bias to a significantly larger degree than prior approaches.
  arXiv  Detail & Related papers  (2020-05-03T02:33:20Z)
• Meta-Learned Confidence for Few-shot Learning [60.6086305523402]
  A popular transductive inference technique for few-shot metric-based approaches, is to update the prototype of each class with the mean of the most confident query examples. We propose to meta-learn the confidence for each query sample, to assign optimal weights to unlabeled queries. We validate our few-shot learning model with meta-learned confidence on four benchmark datasets.
  arXiv  Detail & Related papers  (2020-02-27T10:22:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.