Sense Embeddings are also Biased--Evaluating Social Biases in Static and Contextualised Sense Embeddings

• URL: http://arxiv.org/abs/2203.07523v2
• Date: Wed, 16 Mar 2022 10:22:57 GMT
• Title: Sense Embeddings are also Biased--Evaluating Social Biases in Static and Contextualised Sense Embeddings
• Authors: Yi Zhou, Masahiro Kaneko, Danushka Bollegala
• Abstract summary: One sense of an ambiguous word might be socially biased while its other senses remain unbiased. We create a benchmark dataset for evaluating the social biases in sense embeddings. We propose novel sense-specific bias evaluation measures.
• Score: 28.062567781403274
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Sense embedding learning methods learn different embeddings for the different senses of an ambiguous word. One sense of an ambiguous word might be socially biased while its other senses remain unbiased. In comparison to the numerous prior work evaluating the social biases in pretrained word embeddings, the biases in sense embeddings have been relatively understudied. We create a benchmark dataset for evaluating the social biases in sense embeddings and propose novel sense-specific bias evaluation measures. We conduct an extensive evaluation of multiple static and contextualised sense embeddings for various types of social biases using the proposed measures. Our experimental results show that even in cases where no biases are found at word-level, there still exist worrying levels of social biases at sense-level, which are often ignored by the word-level bias evaluation measures.

Related papers

• Bias in Language Models: Beyond Trick Tests and Toward RUTEd Evaluation [55.66090768926881]
  We study the correspondence between decontextualized "trick tests" and evaluations that are more grounded in Realistic Use and Tangible Effects. We compare three de-contextualized evaluations adapted from the current literature to three analogous RUTEd evaluations applied to long-form content generation. We found no correspondence between trick tests and RUTEd evaluations.
  arXiv  Detail & Related papers  (2024-02-20T01:49:15Z)
• Semantic Properties of cosine based bias scores for word embeddings [48.0753688775574]
  We propose requirements for bias scores to be considered meaningful for quantifying biases. We analyze cosine based scores from the literature with regard to these requirements. We underline these findings with experiments to show that the bias scores' limitations have an impact in the application case.
  arXiv  Detail & Related papers  (2024-01-27T20:31:10Z)
• No Word Embedding Model Is Perfect: Evaluating the Representation Accuracy for Social Bias in the Media [17.4812995898078]
  We study what kind of embedding algorithm serves best to accurately measure types of social bias known to exist in US online news articles. We collect 500k articles and review psychology literature with respect to expected social bias. We compare how models trained with the algorithms on news articles represent the expected social bias.
  arXiv  Detail & Related papers  (2022-11-07T15:45:52Z)
• The Tail Wagging the Dog: Dataset Construction Biases of Social Bias Benchmarks [75.58692290694452]
  We compare social biases with non-social biases stemming from choices made during dataset construction that might not even be discernible to the human eye. We observe that these shallow modifications have a surprising effect on the resulting degree of bias across various models.
  arXiv  Detail & Related papers  (2022-10-18T17:58:39Z)
• Social Biases in Automatic Evaluation Metrics for NLG [53.76118154594404]
  We propose an evaluation method based on Word Embeddings Association Test (WEAT) and Sentence Embeddings Association Test (SEAT) to quantify social biases in evaluation metrics. We construct gender-swapped meta-evaluation datasets to explore the potential impact of gender bias in image caption and text summarization tasks.
  arXiv  Detail & Related papers  (2022-10-17T08:55:26Z)
• The SAME score: Improved cosine based bias score for word embeddings [49.75878234192369]
  We introduce SAME, a novel bias score for semantic bias in embeddings. We show that SAME is capable of measuring semantic bias and identify potential causes for social bias in downstream tasks.
  arXiv  Detail & Related papers  (2022-03-28T09:28:13Z)
• Discovering and Interpreting Biased Concepts in Online Communities [5.670038395203354]
  Language carries implicit human biases, functioning both as a reflection and a perpetuation of stereotypes that people carry with them. ML-based NLP methods such as word embeddings have been shown to learn such language biases with striking accuracy. This paper improves upon, extends, and evaluates our previous data-driven method to automatically discover and help interpret biased concepts encoded in word embeddings.
  arXiv  Detail & Related papers  (2020-10-27T17:07:12Z)
• Towards Debiasing Sentence Representations [109.70181221796469]
  We show that Sent-Debias is effective in removing biases, and at the same time, preserves performance on sentence-level downstream tasks. We hope that our work will inspire future research on characterizing and removing social biases from widely adopted sentence representations for fairer NLP.
  arXiv  Detail & Related papers  (2020-07-16T04:22:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.