Investigating Failures of Automatic Translation in the Case of Unambiguous Gender

• URL: http://arxiv.org/abs/2104.07838v1
• Date: Fri, 16 Apr 2021 00:57:36 GMT
• Title: Investigating Failures of Automatic Translation in the Case of Unambiguous Gender
• Authors: Adithya Renduchintala and Adina Williams
• Abstract summary: Transformer based models are the modern work horses for neural machine translation (NMT) We observe a systemic and rudimentary class of errors made by transformer based models with regards to translating from a language that doesn't mark gender on nouns into others that do. We release an evaluation scheme and dataset for measuring the ability of transformer based NMT models to translate gender correctly.
• Score: 13.58884863186619
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Transformer based models are the modern work horses for neural machine translation (NMT), reaching state of the art across several benchmarks. Despite their impressive accuracy, we observe a systemic and rudimentary class of errors made by transformer based models with regards to translating from a language that doesn't mark gender on nouns into others that do. We find that even when the surrounding context provides unambiguous evidence of the appropriate grammatical gender marking, no transformer based model we tested was able to accurately gender occupation nouns systematically. We release an evaluation scheme and dataset for measuring the ability of transformer based NMT models to translate gender morphology correctly in unambiguous contexts across syntactically diverse sentences. Our dataset translates from an English source into 20 languages from several different language families. With the availability of this dataset, our hope is that the NMT community can iterate on solutions for this class of especially egregious errors.

Related papers

• Generating Gender Alternatives in Machine Translation [13.153018685139413]
  Machine translation systems often translate terms with ambiguous gender into the gendered form that is most prevalent in the systems' training data. This often reflects and perpetuates harmful stereotypes present in society. We study the problem of generating all grammatically correct gendered translation alternatives.
  arXiv  Detail & Related papers  (2024-07-29T22:10:51Z)
• Beyond Binary Gender: Evaluating Gender-Inclusive Machine Translation with Ambiguous Attitude Words [85.48043537327258]
  Existing machine translation gender bias evaluations are primarily focused on male and female genders. This study presents a benchmark AmbGIMT (Gender-Inclusive Machine Translation with Ambiguous attitude words) We propose a novel process to evaluate gender bias based on the Emotional Attitude Score (EAS), which is used to quantify ambiguous attitude words.
  arXiv  Detail & Related papers  (2024-07-23T08:13:51Z)
• Reducing Gender Bias in Machine Translation through Counterfactual Data Generation [0.0]
  We show that gender bias can be significantly mitigated, albeit at the expense of translation quality due to catastrophic forgetting. We also propose a novel domain-adaptation technique that leverages in-domain data created with the counterfactual data generation techniques. The relevant code will be available at Github.
  arXiv  Detail & Related papers  (2023-11-27T23:03:01Z)
• A Tale of Pronouns: Interpretability Informs Gender Bias Mitigation for Fairer Instruction-Tuned Machine Translation [35.44115368160656]
  We investigate whether and to what extent machine translation models exhibit gender bias. We find that IFT models default to male-inflected translations, even disregarding female occupational stereotypes. We propose an easy-to-implement and effective bias mitigation solution.
  arXiv  Detail & Related papers  (2023-10-18T17:36:55Z)
• Unified Model Learning for Various Neural Machine Translation [63.320005222549646]
  Existing machine translation (NMT) studies mainly focus on developing dataset-specific models. We propose a versatile'' model, i.e., the Unified Model Learning for NMT (UMLNMT) that works with data from different tasks. OurNMT results in substantial improvements over dataset-specific models with significantly reduced model deployment costs.
  arXiv  Detail & Related papers  (2023-05-04T12:21:52Z)
• Towards Fine-Grained Information: Identifying the Type and Location of Translation Errors [80.22825549235556]
  Existing approaches can not synchronously consider error position and type. We build an FG-TED model to predict the textbf addition and textbfomission errors. Experiments show that our model can identify both error type and position concurrently, and gives state-of-the-art results.
  arXiv  Detail & Related papers  (2023-02-17T16:20:33Z)
• How sensitive are translation systems to extra contexts? Mitigating gender bias in Neural Machine Translation models through relevant contexts [11.684346035745975]
  A growing number of studies highlight the inherent gender bias that Neural Machine Translation models incorporate during training. We investigate whether these models can be instructed to fix their bias during inference using targeted, guided instructions as contexts. We observe large improvements in reducing the gender bias in translations, across three popular test suites.
  arXiv  Detail & Related papers  (2022-05-22T06:31:54Z)
• When Does Translation Require Context? A Data-driven, Multilingual Exploration [71.43817945875433]
  proper handling of discourse significantly contributes to the quality of machine translation (MT) Recent works in context-aware MT attempt to target a small set of discourse phenomena during evaluation. We develop the Multilingual Discourse-Aware benchmark, a series of taggers that identify and evaluate model performance on discourse phenomena.
  arXiv  Detail & Related papers  (2021-09-15T17:29:30Z)
• ChrEnTranslate: Cherokee-English Machine Translation Demo with Quality Estimation and Corrective Feedback [70.5469946314539]
  ChrEnTranslate is an online machine translation demonstration system for translation between English and an endangered language Cherokee. It supports both statistical and neural translation models as well as provides quality estimation to inform users of reliability.
  arXiv  Detail & Related papers  (2021-07-30T17:58:54Z)
• Decoding and Diversity in Machine Translation [90.33636694717954]
  We characterize differences between cost diversity paid for the BLEU scores enjoyed by NMT. Our study implicates search as a salient source of known bias when translating gender pronouns.
  arXiv  Detail & Related papers  (2020-11-26T21:09:38Z)
• Reducing Gender Bias in Neural Machine Translation as a Domain Adaptation Problem [21.44025591721678]
  Training data for NLP tasks often exhibits gender bias in that fewer sentences refer to women than to men. Recent WinoMT challenge set allows us to measure this effect directly. We use transfer learning on a small set of trusted, gender-balanced examples.
  arXiv  Detail & Related papers  (2020-04-09T11:55:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.