"Thy algorithm shalt not bear false witness": An Evaluation of
Multiclass Debiasing Methods on Word Embeddings
- URL: http://arxiv.org/abs/2010.16228v2
- Date: Wed, 4 Nov 2020 09:24:21 GMT
- Title: "Thy algorithm shalt not bear false witness": An Evaluation of
Multiclass Debiasing Methods on Word Embeddings
- Authors: Thalea Schlender and Gerasimos Spanakis
- Abstract summary: The paper investigates the state-of-the-art multiclass debiasing techniques: Hard debiasing, SoftWEAT debiasing and Conceptor debiasing.
It evaluates their performance when removing religious bias on a common basis by quantifying bias removal via the Word Embedding Association Test (WEAT), Mean Average Cosine Similarity (MAC) and the Relative Negative Sentiment Bias (RNSB)
- Score: 3.0204693431381515
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: With the vast development and employment of artificial intelligence
applications, research into the fairness of these algorithms has been
increased. Specifically, in the natural language processing domain, it has been
shown that social biases persist in word embeddings and are thus in danger of
amplifying these biases when used. As an example of social bias, religious
biases are shown to persist in word embeddings and the need for its removal is
highlighted. This paper investigates the state-of-the-art multiclass debiasing
techniques: Hard debiasing, SoftWEAT debiasing and Conceptor debiasing. It
evaluates their performance when removing religious bias on a common basis by
quantifying bias removal via the Word Embedding Association Test (WEAT), Mean
Average Cosine Similarity (MAC) and the Relative Negative Sentiment Bias
(RNSB). By investigating the religious bias removal on three widely used word
embeddings, namely: Word2Vec, GloVe, and ConceptNet, it is shown that the
preferred method is ConceptorDebiasing. Specifically, this technique manages to
decrease the measured religious bias on average by 82,42%, 96,78% and 54,76%
for the three word embedding sets respectively.
Related papers
- Semantic Properties of cosine based bias scores for word embeddings [52.13994416317707]
We propose requirements for bias scores to be considered meaningful for quantifying biases.
We analyze cosine based scores from the literature with regard to these requirements.
We underline these findings with experiments to show that the bias scores' limitations have an impact in the application case.
arXiv Detail & Related papers (2024-01-27T20:31:10Z) - The SAME score: Improved cosine based bias score for word embeddings [63.24247894974291]
We provide a bias definition based on the ideas from the literature and derive novel requirements for bias scores.
We propose a new bias score, SAME, to address the shortcomings of existing bias scores and show empirically that SAME is better suited to quantify biases in word embeddings.
arXiv Detail & Related papers (2022-03-28T09:28:13Z) - Identification of Biased Terms in News Articles by Comparison of
Outlet-specific Word Embeddings [9.379650501033465]
We train two word embedding models, one on texts of left-wing, the other on right-wing news outlets.
Our hypothesis is that a word's representations in both word embedding spaces are more similar for non-biased words than biased words.
This paper presents the first in-depth look at the context of bias words measured by word embeddings.
arXiv Detail & Related papers (2021-12-14T13:23:49Z) - Evaluating Metrics for Bias in Word Embeddings [64.55554083622258]
We formalize a bias definition based on the ideas from previous works and derive conditions for bias metrics.
We propose a new metric, SAME, to address the shortcomings of existing metrics and mathematically prove that SAME behaves appropriately.
arXiv Detail & Related papers (2021-11-15T16:07:15Z) - Balancing out Bias: Achieving Fairness Through Training Reweighting [58.201275105195485]
Bias in natural language processing arises from models learning characteristics of the author such as gender and race.
Existing methods for mitigating and measuring bias do not directly account for correlations between author demographics and linguistic variables.
This paper introduces a very simple but highly effective method for countering bias using instance reweighting.
arXiv Detail & Related papers (2021-09-16T23:40:28Z) - Towards Debiasing Sentence Representations [109.70181221796469]
We show that Sent-Debias is effective in removing biases, and at the same time, preserves performance on sentence-level downstream tasks.
We hope that our work will inspire future research on characterizing and removing social biases from widely adopted sentence representations for fairer NLP.
arXiv Detail & Related papers (2020-07-16T04:22:30Z) - Nurse is Closer to Woman than Surgeon? Mitigating Gender-Biased
Proximities in Word Embeddings [37.65897382453336]
Existing post-processing methods for debiasing word embeddings are unable to mitigate gender bias hidden in the spatial arrangement of word vectors.
We propose RAN-Debias, a novel gender debiasing methodology which eliminates the bias present in a word vector but also alters the spatial distribution of its neighbouring vectors.
We also propose a new bias evaluation metric - Gender-based Illicit Proximity Estimate (GIPE)
arXiv Detail & Related papers (2020-06-02T20:50:43Z) - Double-Hard Debias: Tailoring Word Embeddings for Gender Bias Mitigation [94.98656228690233]
We propose a technique that purifies the word embeddings against corpus regularities prior to inferring and removing the gender subspace.
Our approach preserves the distributional semantics of the pre-trained word embeddings while reducing gender bias to a significantly larger degree than prior approaches.
arXiv Detail & Related papers (2020-05-03T02:33:20Z) - Joint Multiclass Debiasing of Word Embeddings [5.1135133995376085]
We present a joint multiclass debiasing approach capable of debiasing multiple bias dimensions simultaneously.
We show that our concepts can both reduce or even completely eliminate bias, while maintaining meaningful relationships between vectors in word embeddings.
arXiv Detail & Related papers (2020-03-09T22:06:37Z) - Towards Detection of Subjective Bias using Contextualized Word
Embeddings [9.475039534437332]
We perform experiments for detecting subjective bias using BERT-based models on the Wiki Neutrality Corpus(WNC)
The dataset consists of $360k$ labeled instances, from Wikipedia edits that remove various instances of the bias.
arXiv Detail & Related papers (2020-02-16T18:39:16Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.