ConGA: Guidelines for Contextual Gender Annotation. A Framework for Annotating Gender in Machine Translation

• URL: http://arxiv.org/abs/2603.17962v1
• Date: Wed, 18 Mar 2026 17:31:47 GMT
• Title: ConGA: Guidelines for Contextual Gender Annotation. A Framework for Annotating Gender in Machine Translation
• Abstract summary: Handling gender across languages remains a persistent challenge for Machine Translation.<n>English largely omits grammatical gender, while Italian requires explicit agreement across multiple grammatical categories.<n>We present the Contextual Gender textual (ConGA) framework, a linguistically grounded set of guidelines for word-level gender annotation.
• Score: 0.962211505591938
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Handling gender across languages remains a persistent challenge for Machine Translation (MT) and Large Language Models (LLMs), especially when translating from gender-neutral languages into morphologically gendered ones, such as English to Italian. English largely omits grammatical gender, while Italian requires explicit agreement across multiple grammatical categories. This asymmetry often leads MT systems to default to masculine forms, reinforcing bias and reducing translation accuracy. To address this issue, we present the Contextual Gender Annotation (ConGA) framework, a linguistically grounded set of guidelines for word-level gender annotation. The scheme distinguishes between semantic gender in English through three tags, Masculine (M), Feminine (F), and Ambiguous (A), and grammatical gender realisation in Italian (Masculine (M), Feminine (F)), combined with entity-level identifiers for cross-sentence tracking. We apply ConGA to the gENder-IT dataset, creating a gold-standard resource for evaluating gender bias in translation. Our results reveal systematic masculine overuse and inconsistent feminine realisation, highlighting persistent limitations of current MT systems. By combining fine-grained linguistic annotation with quantitative evaluation, this work offers both a methodology and a benchmark for building more gender-aware and multilingual NLP systems.

Related papers

• Mitigating Gender Bias in English to Romanian Machine Translation [0.6445605125467574]
  Machine translation (MT) systems often fail to correctly translate gender.<n>We propose a hybrid pipeline to mitigate this issue by combining large language model (LLM)-based gender classification with neural machine translation (NMT)<n>Our approach improves gender accuracy on the WinoMT and WinoGender benchmarks by over 40 percentage points compared to a baseline MT system.
  arXiv  Detail & Related papers  (2026-08-09T09:38:59Z)
• Gender Bias in MT for a Genderless Language: New Benchmarks for Basque [1.1419092834574525]
  We introduce two new datasets to evaluate gender bias in translations involving Basque.<n>We evaluate large language models (LLMs) and machine translation (MT) systems.
  arXiv  Detail & Related papers  (2026-03-09T09:35:55Z)
• GAMBIT+: A Challenge Set for Evaluating Gender Bias in Machine Translation Quality Estimation Metrics [18.766033854102663]
  Gender bias in machine translation (MT) systems has been extensively documented, but bias in automatic quality estimation (QE) metrics remains comparatively underexplored.<n>Existing studies suggest that QE metrics can also exhibit gender bias, yet most analyses are limited by small datasets, narrow occupational coverage, and restricted language variety.<n>We introduce a large-scale challenge set specifically designed to probe the behavior of QE metrics when evaluating translations containing gender-ambiguous occupational terms.
  arXiv  Detail & Related papers  (2025-10-08T10:09:03Z)
• EuroGEST: Investigating gender stereotypes in multilingual language models [58.871032460235575]
  We introduce EuroGEST, a dataset designed to measure gender-stereotypical reasoning in LLMs across English and 29 European languages.<n>We show that the strongest stereotypes in all models across all languages are that women are 'beautiful', 'empathetic' and 'neat' and men are 'leaders','strong, tough' and 'professional'
  arXiv  Detail & Related papers  (2025-06-04T11:58:18Z)
• Assumed Identities: Quantifying Gender Bias in Machine Translation of Gender-Ambiguous Occupational Terms [12.568906647547815]
  We introduce GRAPE, a probability-based metric designed to evaluate gender bias.<n>We present GAMBIT, a benchmarking dataset in English with gender-ambiguous occupational terms.<n>Using GRAPE, we evaluate several MT systems and examine whether their gendered translations in Greek and French align with or diverge from societal stereotypes.
  arXiv  Detail & Related papers  (2025-03-06T12:16:14Z)
• Beyond Binary Gender: Evaluating Gender-Inclusive Machine Translation with Ambiguous Attitude Words [85.48043537327258]
  Existing machine translation gender bias evaluations are primarily focused on male and female genders. This study presents a benchmark AmbGIMT (Gender-Inclusive Machine Translation with Ambiguous attitude words) We propose a novel process to evaluate gender bias based on the Emotional Attitude Score (EAS), which is used to quantify ambiguous attitude words.
  arXiv  Detail & Related papers  (2024-07-23T08:13:51Z)
• VisoGender: A dataset for benchmarking gender bias in image-text pronoun resolution [80.57383975987676]
  VisoGender is a novel dataset for benchmarking gender bias in vision-language models. We focus on occupation-related biases within a hegemonic system of binary gender, inspired by Winograd and Winogender schemas. We benchmark several state-of-the-art vision-language models and find that they demonstrate bias in resolving binary gender in complex scenes.
  arXiv  Detail & Related papers  (2023-06-21T17:59:51Z)
• "I'm fully who I am": Towards Centering Transgender and Non-Binary Voices to Measure Biases in Open Language Generation [69.25368160338043]
  Transgender and non-binary (TGNB) individuals disproportionately experience discrimination and exclusion from daily life. We assess how the social reality surrounding experienced marginalization of TGNB persons contributes to and persists within Open Language Generation. We introduce TANGO, a dataset of template-based real-world text curated from a TGNB-oriented community.
  arXiv  Detail & Related papers  (2023-05-17T04:21:45Z)
• Gender Neutralization for an Inclusive Machine Translation: from Theoretical Foundations to Open Challenges [11.37307883423629]
  We explore gender-neutral translation (GNT) as a form of gender inclusivity and a goal to be achieved by machine translation (MT) models. Specifically, we focus on translation from English into Italian, a language pair representative of salient gender-related linguistic transfer problems.
  arXiv  Detail & Related papers  (2023-01-24T15:26:36Z)
• Analyzing Gender Representation in Multilingual Models [59.21915055702203]
  We focus on the representation of gender distinctions as a practical case study. We examine the extent to which the gender concept is encoded in shared subspaces across different languages.
  arXiv  Detail & Related papers  (2022-04-20T00:13:01Z)
• Neural Machine Translation Doesn't Translate Gender Coreference Right Unless You Make It [18.148675498274866]
  We propose schemes for incorporating explicit word-level gender inflection tags into Neural Machine Translation. We find that simple existing approaches can over-generalize a gender-feature to multiple entities in a sentence. We also propose an extension to assess translations of gender-neutral entities from English given a corresponding linguistic convention.
  arXiv  Detail & Related papers  (2020-10-11T20:05:42Z)
• Multi-Dimensional Gender Bias Classification [67.65551687580552]
  Machine learning models can inadvertently learn socially undesirable patterns when training on gender biased text. We propose a general framework that decomposes gender bias in text along several pragmatic and semantic dimensions. Using this fine-grained framework, we automatically annotate eight large scale datasets with gender information.
  arXiv  Detail & Related papers  (2020-05-01T21:23:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.