MORPHOGEN: A Multilingual Benchmark for Evaluating Gender-Aware Morphological Generation

• URL: http://arxiv.org/abs/2604.18914v1
• Date: Mon, 20 Apr 2026 23:35:24 GMT
• Title: MORPHOGEN: A Multilingual Benchmark for Evaluating Gender-Aware Morphological Generation
• Abstract summary: In morphologically rich languages, gender influences verb, pronouns, and even first-person constructions with explicit and implicit mentions of gender.<n>We introduce MORPHOGEN, a morphologically grounded benchmark dataset for evaluating gender-aware generation in three typologically diverse grammatically gendered languages.
• Score: 12.819315569430293
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: While multilingual large language models (LLMs) perform well on high-level tasks like translation and question answering, their ability to handle grammatical gender and morphological agreement remains underexplored. In morphologically rich languages, gender influences verb conjugation, pronouns, and even first-person constructions with explicit and implicit mentions of gender. We introduce MORPHOGEN, a morphologically grounded large-scale benchmark dataset for evaluating gender-aware generation in three typologically diverse grammatically gendered languages: French, Arabic, and Hindi. The core task, GENFORM, requires models to rewrite a first-person sentence in the opposite gender while preserving its meaning and structure. We construct a high-quality synthetic dataset spanning these three languages and benchmark 15 popular multilingual LLMs (2B-70B) on their ability to perform this transformation. Our results reveal significant gaps and interesting insights into how current models handle morphological gender. MORPHOGEN provides a focused diagnostic lens for gender-aware language modeling and lays the groundwork for future research on inclusive and morphology-sensitive NLP.

Related papers

• Gender Bias in MT for a Genderless Language: New Benchmarks for Basque [1.1419092834574525]
  We introduce two new datasets to evaluate gender bias in translations involving Basque.<n>We evaluate large language models (LLMs) and machine translation (MT) systems.
  arXiv  Detail & Related papers  (2026-03-09T09:35:55Z)
• Beyond Content: How Grammatical Gender Shapes Visual Representation in Text-to-Image Models [40.91012720864256]
  We introduce a cross-linguistic benchmark examining words where grammatical gender contradicts stereotypical gender associations.<n>Our dataset spans five gendered languages and two gender-neutral control languages.<n> masculine grammatical markers increase male representation to 73% on average, while feminine grammatical markers increase female representation to 38%.
  arXiv  Detail & Related papers  (2025-08-05T08:13:07Z)
• EuroGEST: Investigating gender stereotypes in multilingual language models [58.871032460235575]
  We introduce EuroGEST, a dataset designed to measure gender-stereotypical reasoning in LLMs across English and 29 European languages.<n>We show that the strongest stereotypes in all models across all languages are that women are 'beautiful', 'empathetic' and 'neat' and men are 'leaders','strong, tough' and 'professional'
  arXiv  Detail & Related papers  (2025-06-04T11:58:18Z)
• Mind the Inclusivity Gap: Multilingual Gender-Neutral Translation Evaluation with mGeNTE [34.11872938329087]
  Gender-neutral translation (GNT) represents a linguistic strategy towards fairer communication across languages.<n>We introduce mGeNTE, an expert-curated resource, and use it to conduct the first systematic multilingual evaluation of inclusive translation.<n>Experiments on en-es/de/it/el reveal that while models can recognize when neutrality is appropriate, they cannot consistently produce neutral translations.
  arXiv  Detail & Related papers  (2025-01-16T09:35:15Z)
• The Lou Dataset -- Exploring the Impact of Gender-Fair Language in German Text Classification [57.06913662622832]
  Gender-fair language fosters inclusion by addressing all genders or using neutral forms. Gender-fair language substantially impacts predictions by flipping labels, reducing certainty, and altering attention patterns. While we offer initial insights on the effect on German text classification, the findings likely apply to other languages.
  arXiv  Detail & Related papers  (2024-09-26T15:08:17Z)
• What an Elegant Bridge: Multilingual LLMs are Biased Similarly in Different Languages [51.0349882045866]
  This paper investigates biases of Large Language Models (LLMs) through the lens of grammatical gender. We prompt a model to describe nouns with adjectives in various languages, focusing specifically on languages with grammatical gender. We find that a simple classifier can not only predict noun gender above chance but also exhibit cross-language transferability.
  arXiv  Detail & Related papers  (2024-07-12T22:10:16Z)
• Leveraging Large Language Models to Measure Gender Representation Bias in Gendered Language Corpora [9.959039325564744]
  Large language models (LLMs) often inherit and amplify social biases embedded in their training data.<n>Gender bias is the association of specific roles or traits with a particular gender.<n>Gender representation bias is the unequal frequency of references to individuals of different genders.
  arXiv  Detail & Related papers  (2024-06-19T16:30:58Z)
• Gender Bias in Large Language Models across Multiple Languages [10.068466432117113]
  We examine gender bias in large language models (LLMs) generated for different languages. We use three measurements: 1) gender bias in selecting descriptive words given the gender-related context. 2) gender bias in selecting gender-related pronouns (she/he) given the descriptive words.
  arXiv  Detail & Related papers  (2024-03-01T04:47:16Z)
• Multilingual Text-to-Image Generation Magnifies Gender Stereotypes and Prompt Engineering May Not Help You [64.74707085021858]
  We show that multilingual models suffer from significant gender biases just as monolingual models do.<n>We propose a novel benchmark, MAGBIG, intended to foster research on gender bias in multilingual models.<n>Our results show that not only do models exhibit strong gender biases but they also behave differently across languages.
  arXiv  Detail & Related papers  (2024-01-29T12:02:28Z)
• Using Artificial French Data to Understand the Emergence of Gender Bias in Transformer Language Models [5.22145960878624]
  This work takes an initial step towards exploring the less researched topic of how neural models discover linguistic properties of words, such as gender, as well as the rules governing their usage. We propose to use an artificial corpus generated by a PCFG based on French to precisely control the gender distribution in the training data and determine under which conditions a model correctly captures gender information or, on the contrary, appears gender-biased.
  arXiv  Detail & Related papers  (2023-10-24T14:08:37Z)
• Analyzing Gender Representation in Multilingual Models [59.21915055702203]
  We focus on the representation of gender distinctions as a practical case study. We examine the extent to which the gender concept is encoded in shared subspaces across different languages.
  arXiv  Detail & Related papers  (2022-04-20T00:13:01Z)
• Under the Morphosyntactic Lens: A Multifaceted Evaluation of Gender Bias in Speech Translation [20.39599469927542]
  Gender bias is largely recognized as a problematic phenomenon affecting language technologies. Most of current evaluation practices adopt a word-level focus on a narrow set of occupational nouns under synthetic conditions. Such protocols overlook key features of grammatical gender languages, which are characterized by morphosyntactic chains of gender agreement.
  arXiv  Detail & Related papers  (2022-03-18T11:14:16Z)
• AM2iCo: Evaluating Word Meaning in Context across Low-ResourceLanguages with Adversarial Examples [51.048234591165155]
  We present AM2iCo, Adversarial and Multilingual Meaning in Context. It aims to faithfully assess the ability of state-of-the-art (SotA) representation models to understand the identity of word meaning in cross-lingual contexts. Results reveal that current SotA pretrained encoders substantially lag behind human performance.
  arXiv  Detail & Related papers  (2021-04-17T20:23:45Z)
• Quantifying Gender Bias Towards Politicians in Cross-Lingual Language Models [104.41668491794974]
  We quantify the usage of adjectives and verbs generated by language models surrounding the names of politicians as a function of their gender. We find that while some words such as dead, and designated are associated with both male and female politicians, a few specific words such as beautiful and divorced are predominantly associated with female politicians.
  arXiv  Detail & Related papers  (2021-04-15T15:03:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.