Transforming Dutch: Debiasing Dutch Coreference Resolution Systems for Non-binary Pronouns

• URL: http://arxiv.org/abs/2405.00134v1
• Date: Tue, 30 Apr 2024 18:31:19 GMT
• Title: Transforming Dutch: Debiasing Dutch Coreference Resolution Systems for Non-binary Pronouns
• Authors: Goya van Boven, Yupei Du, Dong Nguyen,
• Abstract summary: Gender-neutral pronouns are increasingly being introduced across Western languages. Recent evaluations have demonstrated that English NLP systems are unable to correctly process gender-neutral pronouns. This paper examines a Dutch coreference resolution system's performance on gender-neutral pronouns.
• Score: 5.5514102920271196
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Gender-neutral pronouns are increasingly being introduced across Western languages. Recent evaluations have however demonstrated that English NLP systems are unable to correctly process gender-neutral pronouns, with the risk of erasing and misgendering non-binary individuals. This paper examines a Dutch coreference resolution system's performance on gender-neutral pronouns, specifically hen and die. In Dutch, these pronouns were only introduced in 2016, compared to the longstanding existence of singular they in English. We additionally compare two debiasing techniques for coreference resolution systems in non-binary contexts: Counterfactual Data Augmentation (CDA) and delexicalisation. Moreover, because pronoun performance can be hard to interpret from a general evaluation metric like LEA, we introduce an innovative evaluation metric, the pronoun score, which directly represents the portion of correctly processed pronouns. Our results reveal diminished performance on gender-neutral pronouns compared to gendered counterparts. Nevertheless, although delexicalisation fails to yield improvements, CDA substantially reduces the performance gap between gendered and gender-neutral pronouns. We further show that CDA remains effective in low-resource settings, in which a limited set of debiasing documents is used. This efficacy extends to previously unseen neopronouns, which are currently infrequently used but may gain popularity in the future, underscoring the viability of effective debiasing with minimal resources and low computational costs.

Related papers

• GenderCARE: A Comprehensive Framework for Assessing and Reducing Gender Bias in Large Language Models [73.23743278545321]
  Large language models (LLMs) have exhibited remarkable capabilities in natural language generation, but have also been observed to magnify societal biases. GenderCARE is a comprehensive framework that encompasses innovative Criteria, bias Assessment, Reduction techniques, and Evaluation metrics.
  arXiv  Detail & Related papers  (2024-08-22T15:35:46Z)
• Beyond Binary Gender: Evaluating Gender-Inclusive Machine Translation with Ambiguous Attitude Words [85.48043537327258]
  Existing machine translation gender bias evaluations are primarily focused on male and female genders. This study presents a benchmark AmbGIMT (Gender-Inclusive Machine Translation with Ambiguous attitude words) We propose a novel process to evaluate gender bias based on the Emotional Attitude Score (EAS), which is used to quantify ambiguous attitude words.
  arXiv  Detail & Related papers  (2024-07-23T08:13:51Z)
• Tokenization Matters: Navigating Data-Scarce Tokenization for Gender Inclusive Language Technologies [75.85462924188076]
  Gender-inclusive NLP research has documented the harmful limitations of gender binary-centric large language models (LLM) We find that misgendering is significantly influenced by Byte-Pair (BPE) tokenization. We propose two techniques: (1) pronoun tokenization parity, a method to enforce consistent tokenization across gendered pronouns, and (2) utilizing pre-existing LLM pronoun knowledge to improve neopronoun proficiency.
  arXiv  Detail & Related papers  (2023-12-19T01:28:46Z)
• VisoGender: A dataset for benchmarking gender bias in image-text pronoun resolution [80.57383975987676]
  VisoGender is a novel dataset for benchmarking gender bias in vision-language models. We focus on occupation-related biases within a hegemonic system of binary gender, inspired by Winograd and Winogender schemas. We benchmark several state-of-the-art vision-language models and find that they demonstrate bias in resolving binary gender in complex scenes.
  arXiv  Detail & Related papers  (2023-06-21T17:59:51Z)
• MISGENDERED: Limits of Large Language Models in Understanding Pronouns [46.276320374441056]
  We evaluate popular language models for their ability to correctly use English gender-neutral pronouns. We introduce MISGENDERED, a framework for evaluating large language models' ability to correctly use preferred pronouns.
  arXiv  Detail & Related papers  (2023-06-06T18:27:52Z)
• What about em? How Commercial Machine Translation Fails to Handle (Neo-)Pronouns [26.28827649737955]
  Wrong pronoun translations can discriminate against marginalized groups, e.g., non-binary individuals. We study how three commercial machine translation systems translate 3rd-person pronouns. Our error analysis shows that the presence of a gender-neutral pronoun often leads to grammatical and semantic translation errors.
  arXiv  Detail & Related papers  (2023-05-25T13:34:09Z)
• "I'm fully who I am": Towards Centering Transgender and Non-Binary Voices to Measure Biases in Open Language Generation [69.25368160338043]
  Transgender and non-binary (TGNB) individuals disproportionately experience discrimination and exclusion from daily life. We assess how the social reality surrounding experienced marginalization of TGNB persons contributes to and persists within Open Language Generation. We introduce TANGO, a dataset of template-based real-world text curated from a TGNB-oriented community.
  arXiv  Detail & Related papers  (2023-05-17T04:21:45Z)
• How Conservative are Language Models? Adapting to the Introduction of Gender-Neutral Pronouns [0.15293427903448023]
  We show that gender-neutral pronouns (in Swedish) are not associated with human processing difficulties. We show that gender-neutral pronouns in Danish, English, and Swedish are associated with higher perplexity, more dispersed attention patterns, and worse downstream performance.
  arXiv  Detail & Related papers  (2022-04-11T09:42:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.