Gender Disambiguation in Machine Translation: Diagnostic Evaluation in Decoder-Only Architectures

• URL: http://arxiv.org/abs/2603.17952v1
• Date: Wed, 18 Mar 2026 17:26:36 GMT
• Title: Gender Disambiguation in Machine Translation: Diagnostic Evaluation in Decoder-Only Architectures
• Abstract summary: We introduce a novel measure coined "Prior Bias", capturing a model's default gender assumptions.<n>We show that, despite their scale and state-of-the-art status, decoder-only models do not generally outperform encoder-decoder architectures on gender-specific metrics.
• Score: 5.038281415151629
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While Large Language Models achieve state-of-the-art results across a wide range of NLP tasks, they remain prone to systematic biases. Among these, gender bias is particularly salient in MT, due to systematic differences across languages in whether and how gender is marked. As a result, translation often requires disambiguating implicit source signals into explicit gender-marked forms. In this context, standard benchmarks may capture broad disparities but fail to reflect the full complexity of gender bias in modern MT. In this paper, we extend recent frameworks on bias evaluation by: (i) introducing a novel measure coined "Prior Bias", capturing a model's default gender assumptions, and (ii) applying the framework to decoder-only MT models. Our results show that, despite their scale and state-of-the-art status, decoder-only models do not generally outperform encoder-decoder architectures on gender-specific metrics; however, post-training (e.g., instruction tuning) not only improves contextual awareness but also reduces the masculine Prior Bias.

Related papers

• Is She Even Relevant? When BERT Ignores Explicit Gender Cues [0.962211505591938]
  This paper investigates how and when gender information emerges in a Dutch BERT model trained from scratch.<n>It offers one of the first checkpoint-level analyses of bias formation in a Transformer architecture for a language combining overt morphological gender marking and generic forms.
  arXiv  Detail & Related papers  (2026-05-08T11:48:22Z)
• Alignment Reduces Expressed but Not Encoded Gender Bias: A Unified Framework and Study [3.679036235271287]
  We propose a unified framework to jointly analyze intrinsic and extrinsic gender bias in Large Language Models.<n>We find a consistent association between latent gender information and expressed bias when measured under the unified protocol.<n>Our results suggest that while the latter indeed reduces expressed bias, measurable gender-related associations are still present in internal representations.
  arXiv  Detail & Related papers  (2026-03-25T09:35:18Z)
• Are We Paying Attention to Her? Investigating Gender Disambiguation and Attention in Machine Translation [4.881426374773398]
  We propose a novel evaluation metric called Minimal Pair Accuracy (MPA)<n>MPA focuses on whether models adapt to gender cues in minimal pairs.<n>MPA shows that in anti-stereotypical cases, NMT models tend to more consistently take masculine gender cues into account.
  arXiv  Detail & Related papers  (2025-05-13T13:17:23Z)
• Assumed Identities: Quantifying Gender Bias in Machine Translation of Gender-Ambiguous Occupational Terms [12.568906647547815]
  We introduce GRAPE, a probability-based metric designed to evaluate gender bias.<n>We present GAMBIT, a benchmarking dataset in English with gender-ambiguous occupational terms.<n>Using GRAPE, we evaluate several MT systems and examine whether their gendered translations in Greek and French align with or diverge from societal stereotypes.
  arXiv  Detail & Related papers  (2025-03-06T12:16:14Z)
• GenderCARE: A Comprehensive Framework for Assessing and Reducing Gender Bias in Large Language Models [73.23743278545321]
  Large language models (LLMs) have exhibited remarkable capabilities in natural language generation, but have also been observed to magnify societal biases.<n>GenderCARE is a comprehensive framework that encompasses innovative Criteria, bias Assessment, Reduction techniques, and Evaluation metrics.
  arXiv  Detail & Related papers  (2024-08-22T15:35:46Z)
• Beyond Binary Gender: Evaluating Gender-Inclusive Machine Translation with Ambiguous Attitude Words [85.48043537327258]
  Existing machine translation gender bias evaluations are primarily focused on male and female genders. This study presents a benchmark AmbGIMT (Gender-Inclusive Machine Translation with Ambiguous attitude words) We propose a novel process to evaluate gender bias based on the Emotional Attitude Score (EAS), which is used to quantify ambiguous attitude words.
  arXiv  Detail & Related papers  (2024-07-23T08:13:51Z)
• A Tale of Pronouns: Interpretability Informs Gender Bias Mitigation for Fairer Instruction-Tuned Machine Translation [35.44115368160656]
  We investigate whether and to what extent machine translation models exhibit gender bias. We find that IFT models default to male-inflected translations, even disregarding female occupational stereotypes. We propose an easy-to-implement and effective bias mitigation solution.
  arXiv  Detail & Related papers  (2023-10-18T17:36:55Z)
• Counter-GAP: Counterfactual Bias Evaluation through Gendered Ambiguous Pronouns [53.62845317039185]
  Bias-measuring datasets play a critical role in detecting biased behavior of language models. We propose a novel method to collect diverse, natural, and minimally distant text pairs via counterfactual generation. We show that four pre-trained language models are significantly more inconsistent across different gender groups than within each group.
  arXiv  Detail & Related papers  (2023-02-11T12:11:03Z)
• Decoding and Diversity in Machine Translation [90.33636694717954]
  We characterize differences between cost diversity paid for the BLEU scores enjoyed by NMT. Our study implicates search as a salient source of known bias when translating gender pronouns.
  arXiv  Detail & Related papers  (2020-11-26T21:09:38Z)
• Gender Stereotype Reinforcement: Measuring the Gender Bias Conveyed by Ranking Algorithms [68.85295025020942]
  We propose the Gender Stereotype Reinforcement (GSR) measure, which quantifies the tendency of a Search Engines to support gender stereotypes. GSR is the first specifically tailored measure for Information Retrieval, capable of quantifying representational harms.
  arXiv  Detail & Related papers  (2020-09-02T20:45:04Z)
• Multi-Dimensional Gender Bias Classification [67.65551687580552]
  Machine learning models can inadvertently learn socially undesirable patterns when training on gender biased text. We propose a general framework that decomposes gender bias in text along several pragmatic and semantic dimensions. Using this fine-grained framework, we automatically annotate eight large scale datasets with gender information.
  arXiv  Detail & Related papers  (2020-05-01T21:23:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.