Evaluating Machine Translation Datasets for Low-Web Data Languages: A Gendered Lens

• URL: http://arxiv.org/abs/2511.03880v1
• Date: Wed, 05 Nov 2025 21:51:30 GMT
• Title: Evaluating Machine Translation Datasets for Low-Web Data Languages: A Gendered Lens
• Abstract summary: In prioritizing quantity over quality, we risk building language technologies that perform poorly for low-resourced languages.<n>In this paper, we investigate the quality of Machine Translation datasets for three low-resourced languages.<n>We found a large skew towards the male gender--in names of persons, the grammatical gender of verbs, and in stereotypical depictions in the datasets.
• Score: 2.0009620796960523
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: As low-resourced languages are increasingly incorporated into NLP research, there is an emphasis on collecting large-scale datasets. But in prioritizing quantity over quality, we risk 1) building language technologies that perform poorly for these languages and 2) producing harmful content that perpetuates societal biases. In this paper, we investigate the quality of Machine Translation (MT) datasets for three low-resourced languages--Afan Oromo, Amharic, and Tigrinya, with a focus on the gender representation in the datasets. Our findings demonstrate that while training data has a large representation of political and religious domain text, benchmark datasets are focused on news, health, and sports. We also found a large skew towards the male gender--in names of persons, the grammatical gender of verbs, and in stereotypical depictions in the datasets. Further, we found harmful and toxic depictions against women, which were more prominent for the language with the largest amount of data, underscoring that quantity does not guarantee quality. We hope that our work inspires further inquiry into the datasets collected for low-resourced languages and prompts early mitigation of harmful content. WARNING: This paper contains discussion of NSFW content that some may find disturbing.

Related papers

• EuroGEST: Investigating gender stereotypes in multilingual language models [58.871032460235575]
  We introduce EuroGEST, a dataset designed to measure gender-stereotypical reasoning in LLMs across English and 29 European languages.<n>We show that the strongest stereotypes in all models across all languages are that women are 'beautiful', 'empathetic' and 'neat' and men are 'leaders','strong, tough' and 'professional'
  arXiv  Detail & Related papers  (2025-06-04T11:58:18Z)
• Beyond Binary Gender: Evaluating Gender-Inclusive Machine Translation with Ambiguous Attitude Words [85.48043537327258]
  Existing machine translation gender bias evaluations are primarily focused on male and female genders. This study presents a benchmark AmbGIMT (Gender-Inclusive Machine Translation with Ambiguous attitude words) We propose a novel process to evaluate gender bias based on the Emotional Attitude Score (EAS), which is used to quantify ambiguous attitude words.
  arXiv  Detail & Related papers  (2024-07-23T08:13:51Z)
• Multilingual Diversity Improves Vision-Language Representations [97.16233528393356]
  Pre-training on this dataset outperforms using English-only or English-dominated datasets on ImageNet.<n>On a geographically diverse task like GeoDE, we also observe improvements across all regions, with the biggest gain coming from Africa.
  arXiv  Detail & Related papers  (2024-05-27T08:08:51Z)
• Multilingual Text-to-Image Generation Magnifies Gender Stereotypes and Prompt Engineering May Not Help You [64.74707085021858]
  We show that multilingual models suffer from significant gender biases just as monolingual models do.<n>We propose a novel benchmark, MAGBIG, intended to foster research on gender bias in multilingual models.<n>Our results show that not only do models exhibit strong gender biases but they also behave differently across languages.
  arXiv  Detail & Related papers  (2024-01-29T12:02:28Z)
• Misgendering and Assuming Gender in Machine Translation when Working with Low-Resource Languages [0.0]
  This chapter focuses on gender-related errors in machine translation (MT) in the context of low-resource languages. We begin by explaining what low-resource languages are, examining the inseparable social and computational factors that create such linguistic hierarchies. We discuss the postcolonial and societal impacts of such errors leading to linguistic erasure and representational harms.
  arXiv  Detail & Related papers  (2024-01-24T00:58:30Z)
• MiTTenS: A Dataset for Evaluating Gender Mistranslation [15.33020859049458]
  We introduce a dataset, MiTTenS, covering 26 languages from a variety of language families and scripts. The dataset is constructed with handcrafted passages that target known failure patterns. We show that all systems exhibit gender mistranslation and potential harm, even in high resource languages.
  arXiv  Detail & Related papers  (2024-01-13T00:08:23Z)
• NusaWrites: Constructing High-Quality Corpora for Underrepresented and Extremely Low-Resource Languages [54.808217147579036]
  We conduct a case study on Indonesian local languages. We compare the effectiveness of online scraping, human translation, and paragraph writing by native speakers in constructing datasets. Our findings demonstrate that datasets generated through paragraph writing by native speakers exhibit superior quality in terms of lexical diversity and cultural content.
  arXiv  Detail & Related papers  (2023-09-19T14:42:33Z)
• No Language Left Behind: Scaling Human-Centered Machine Translation [69.28110770760506]
  We create datasets and models aimed at narrowing the performance gap between low and high-resource languages. We propose multiple architectural and training improvements to counteract overfitting while training on thousands of tasks. Our model achieves an improvement of 44% BLEU relative to the previous state-of-the-art.
  arXiv  Detail & Related papers  (2022-07-11T07:33:36Z)
• COLD: A Benchmark for Chinese Offensive Language Detection [54.60909500459201]
  We use COLDataset, a Chinese offensive language dataset with 37k annotated sentences. We also propose textscCOLDetector to study output offensiveness of popular Chinese language models. Our resources and analyses are intended to help detoxify the Chinese online communities and evaluate the safety performance of generative language models.
  arXiv  Detail & Related papers  (2022-01-16T11:47:23Z)
• SWSR: A Chinese Dataset and Lexicon for Online Sexism Detection [9.443571652110663]
  We propose the first Chinese sexism dataset -- Sina Weibo Sexism Review (SWSR) dataset -- and a large Chinese lexicon SexHateLex. SWSR dataset provides labels at different levels of granularity including (i) sexism or non-sexism, (ii) sexism category and (iii) target type. We conduct experiments for the three sexism classification tasks making use of state-of-the-art machine learning models.
  arXiv  Detail & Related papers  (2021-08-06T12:06:40Z)
• Gender in Danger? Evaluating Speech Translation Technology on the MuST-SHE Corpus [20.766890957411132]
  Translating from languages without productive grammatical gender like English into gender-marked languages is a well-known difficulty for machines. Can audio provide additional information to reduce gender bias? We present the first thorough investigation of gender bias in speech translation, contributing with the release of a benchmark useful for future studies.
  arXiv  Detail & Related papers  (2020-06-10T09:55:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.