GATE: A Challenge Set for Gender-Ambiguous Translation Examples

• URL: http://arxiv.org/abs/2303.03975v1
• Date: Tue, 7 Mar 2023 15:23:38 GMT
• Title: GATE: A Challenge Set for Gender-Ambiguous Translation Examples
• Authors: Spencer Rarrick, Ranjita Naik, Varun Mathur, Sundar Poudel, Vishal Chowdhary
• Abstract summary: When source gender is ambiguous, machine translation models typically default to stereotypical gender roles, perpetuating harmful bias. Recent work has led to the development of "gender rewriters" that generate alternative gender translations on such ambiguous inputs, but such systems are plagued by poor linguistic coverage. We present and release GATE, a linguistically diverse corpus of gender-ambiguous source sentences along with multiple alternative target language translations.
• Score: 0.31498833540989407
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Although recent years have brought significant progress in improving translation of unambiguously gendered sentences, translation of ambiguously gendered input remains relatively unexplored. When source gender is ambiguous, machine translation models typically default to stereotypical gender roles, perpetuating harmful bias. Recent work has led to the development of "gender rewriters" that generate alternative gender translations on such ambiguous inputs, but such systems are plagued by poor linguistic coverage. To encourage better performance on this task we present and release GATE, a linguistically diverse corpus of gender-ambiguous source sentences along with multiple alternative target language translations. We also provide tools for evaluation and system analysis when using GATE and use them to evaluate our translation rewriter system.

Related papers

• Beyond Binary Gender: Evaluating Gender-Inclusive Machine Translation with Ambiguous Attitude Words [85.48043537327258]
  Existing machine translation gender bias evaluations are primarily focused on male and female genders. This study presents a benchmark AmbGIMT (Gender-Inclusive Machine Translation with Ambiguous attitude words) We propose a novel process to evaluate gender bias based on the Emotional Attitude Score (EAS), which is used to quantify ambiguous attitude words.
  arXiv  Detail & Related papers  (2024-07-23T08:13:51Z)
• GATE X-E : A Challenge Set for Gender-Fair Translations from Weakly-Gendered Languages [0.0]
  We introduce GATE X-E, an extension to the GATE corpus, that consists of human translations from Turkish, Hungarian, Finnish, and Persian into English. The dataset features natural sentences with a wide range of sentence lengths and domains, challenging translation rewriters on various linguistic phenomena. We present a translation gender rewriting solution built with GPT-4 and use GATE X-E to evaluate it.
  arXiv  Detail & Related papers  (2024-02-22T04:36:14Z)
• Evaluating Gender Bias in the Translation of Gender-Neutral Languages into English [0.0]
  We introduce GATE X-E, an extension to the GATE corpus, that consists of human translations from Turkish, Hungarian, Finnish, and Persian into English. The dataset features natural sentences with a wide range of sentence lengths and domains, challenging translation rewriters on various linguistic phenomena. We present an English gender rewriting solution built on GPT-3.5 Turbo and use GATE X-E to evaluate it.
  arXiv  Detail & Related papers  (2023-11-15T10:25:14Z)
• The Gender-GAP Pipeline: A Gender-Aware Polyglot Pipeline for Gender Characterisation in 55 Languages [51.2321117760104]
  This paper describes the Gender-GAP Pipeline, an automatic pipeline to characterize gender representation in large-scale datasets for 55 languages. The pipeline uses a multilingual lexicon of gendered person-nouns to quantify the gender representation in text. We showcase it to report gender representation in WMT training data and development data for the News task, confirming that current data is skewed towards masculine representation.
  arXiv  Detail & Related papers  (2023-08-31T17:20:50Z)
• VisoGender: A dataset for benchmarking gender bias in image-text pronoun resolution [80.57383975987676]
  VisoGender is a novel dataset for benchmarking gender bias in vision-language models. We focus on occupation-related biases within a hegemonic system of binary gender, inspired by Winograd and Winogender schemas. We benchmark several state-of-the-art vision-language models and find that they demonstrate bias in resolving binary gender in complex scenes.
  arXiv  Detail & Related papers  (2023-06-21T17:59:51Z)
• Target-Agnostic Gender-Aware Contrastive Learning for Mitigating Bias in Multilingual Machine Translation [28.471506840241602]
  Gender bias is a significant issue in machine translation, leading to ongoing research efforts in developing bias mitigation techniques. We propose a bias mitigation method based on a novel approach. Gender-Aware Contrastive Learning, GACL, encodes contextual gender information into the representations of non-explicit gender words.
  arXiv  Detail & Related papers  (2023-05-23T12:53:39Z)
• Towards Understanding Gender-Seniority Compound Bias in Natural Language Generation [64.65911758042914]
  We investigate how seniority impacts the degree of gender bias exhibited in pretrained neural generation models. Our results show that GPT-2 amplifies bias by considering women as junior and men as senior more often than the ground truth in both domains. These results suggest that NLP applications built using GPT-2 may harm women in professional capacities.
  arXiv  Detail & Related papers  (2022-05-19T20:05:02Z)
• Improving Gender Translation Accuracy with Filtered Self-Training [14.938401898546548]
  Machine translation systems often output incorrect gender, even when the gender is clear from context. We propose a gender-filtered self-training technique to improve gender translation accuracy on unambiguously gendered inputs.
  arXiv  Detail & Related papers  (2021-04-15T18:05:29Z)
• They, Them, Theirs: Rewriting with Gender-Neutral English [56.14842450974887]
  We perform a case study on the singular they, a common way to promote gender inclusion in English. We show how a model can be trained to produce gender-neutral English with 1% word error rate with no human-labeled data.
  arXiv  Detail & Related papers  (2021-02-12T21:47:48Z)
• Multi-Dimensional Gender Bias Classification [67.65551687580552]
  Machine learning models can inadvertently learn socially undesirable patterns when training on gender biased text. We propose a general framework that decomposes gender bias in text along several pragmatic and semantic dimensions. Using this fine-grained framework, we automatically annotate eight large scale datasets with gender information.
  arXiv  Detail & Related papers  (2020-05-01T21:23:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.