Using Artificial French Data to Understand the Emergence of Gender Bias in Transformer Language Models

• URL: http://arxiv.org/abs/2310.15852v1
• Date: Tue, 24 Oct 2023 14:08:37 GMT
• Title: Using Artificial French Data to Understand the Emergence of Gender Bias in Transformer Language Models
• Authors: Lina Conti and Guillaume Wisniewski
• Abstract summary: This work takes an initial step towards exploring the less researched topic of how neural models discover linguistic properties of words, such as gender, as well as the rules governing their usage. We propose to use an artificial corpus generated by a PCFG based on French to precisely control the gender distribution in the training data and determine under which conditions a model correctly captures gender information or, on the contrary, appears gender-biased.
• Score: 5.22145960878624
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Numerous studies have demonstrated the ability of neural language models to learn various linguistic properties without direct supervision. This work takes an initial step towards exploring the less researched topic of how neural models discover linguistic properties of words, such as gender, as well as the rules governing their usage. We propose to use an artificial corpus generated by a PCFG based on French to precisely control the gender distribution in the training data and determine under which conditions a model correctly captures gender information or, on the contrary, appears gender-biased.

Related papers

• Leveraging Large Language Models to Measure Gender Bias in Gendered Languages [9.959039325564744]
  This paper introduces a novel methodology that leverages the contextual understanding capabilities of large language models (LLMs) to quantitatively analyze gender representation in Spanish corpora. We empirically validate our method on four widely-used benchmark datasets, uncovering significant gender disparities with a male-to-female ratio ranging from 4:01.
  arXiv  Detail & Related papers  (2024-06-19T16:30:58Z)
• Learning Phonotactics from Linguistic Informants [54.086544221761486]
  Our model iteratively selects or synthesizes a data-point according to one of a range of information-theoretic policies. We find that the information-theoretic policies that our model uses to select items to query the informant achieve sample efficiency comparable to, or greater than, fully supervised approaches.
  arXiv  Detail & Related papers  (2024-05-08T00:18:56Z)
• Investigating grammatical abstraction in language models using few-shot learning of novel noun gender [0.0]
  We conduct a noun learning experiment to assess whether an LSTM and a decoder-only transformer can achieve human-like abstraction of grammatical gender in French. We find that both language models effectively generalise novel noun gender from one to two learning examples and apply the learnt gender across agreement contexts. While the generalisation behaviour of models suggests that they represent grammatical gender as an abstract category, like humans, further work is needed to explore the details.
  arXiv  Detail & Related papers  (2024-03-15T14:25:59Z)
• Transparency at the Source: Evaluating and Interpreting Language Models With Access to the True Distribution [4.01799362940916]
  We present a setup for training, evaluating and interpreting neural language models, that uses artificial, language-like data. The data is generated using a massive probabilistic grammar, that is itself derived from a large natural language corpus. With access to the underlying true source, our results show striking differences and outcomes in learning dynamics between different classes of words.
  arXiv  Detail & Related papers  (2023-10-23T12:03:01Z)
• Exploiting Biased Models to De-bias Text: A Gender-Fair Rewriting Model [32.21372089380992]
  We train a rewriting model for German without the need for elaborate handcrafted rules. The outputs of this model increased gender-fairness as shown in a human evaluation study.
  arXiv  Detail & Related papers  (2023-05-18T17:35:28Z)
• The Birth of Bias: A case study on the evolution of gender bias in an English language model [1.6344851071810076]
  We use a relatively small language model, using the LSTM architecture trained on an English Wikipedia corpus. We find that the representation of gender is dynamic and identify different phases during training. We show that gender information is represented increasingly locally in the input embeddings of the model.
  arXiv  Detail & Related papers  (2022-07-21T00:59:04Z)
• Naturalistic Causal Probing for Morpho-Syntax [76.83735391276547]
  We suggest a naturalistic strategy for input-level intervention on real world data in Spanish. Using our approach, we isolate morpho-syntactic features from counfounders in sentences. We apply this methodology to analyze causal effects of gender and number on contextualized representations extracted from pre-trained models.
  arXiv  Detail & Related papers  (2022-05-14T11:47:58Z)
• Analyzing Gender Representation in Multilingual Models [59.21915055702203]
  We focus on the representation of gender distinctions as a practical case study. We examine the extent to which the gender concept is encoded in shared subspaces across different languages.
  arXiv  Detail & Related papers  (2022-04-20T00:13:01Z)
• Improving Gender Fairness of Pre-Trained Language Models without Catastrophic Forgetting [88.83117372793737]
  Forgetting information in the original training data may damage the model's downstream performance by a large margin. We propose GEnder Equality Prompt (GEEP) to improve gender fairness of pre-trained models with less forgetting.
  arXiv  Detail & Related papers  (2021-10-11T15:52:16Z)
• Towards Zero-shot Language Modeling [90.80124496312274]
  We construct a neural model that is inductively biased towards learning human languages. We infer this distribution from a sample of typologically diverse training languages. We harness additional language-specific side information as distant supervision for held-out languages.
  arXiv  Detail & Related papers  (2021-08-06T23:49:18Z)
• Multi-Dimensional Gender Bias Classification [67.65551687580552]
  Machine learning models can inadvertently learn socially undesirable patterns when training on gender biased text. We propose a general framework that decomposes gender bias in text along several pragmatic and semantic dimensions. Using this fine-grained framework, we automatically annotate eight large scale datasets with gender information.
  arXiv  Detail & Related papers  (2020-05-01T21:23:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.