MiTTenS: A Dataset for Evaluating Misgendering in Translation
- URL: http://arxiv.org/abs/2401.06935v1
- Date: Sat, 13 Jan 2024 00:08:23 GMT
- Title: MiTTenS: A Dataset for Evaluating Misgendering in Translation
- Authors: Kevin Robinson, Sneha Kudugunta, Romina Stella, Sunipa Dev, Jasmijn
Bastings
- Abstract summary: Misgendering is the act of referring to someone in a way that does not reflect their gender identity.
We introduce a dataset, MiTTenS, covering 26 languages from a variety of language families and scripts.
- Score: 16.446952262028358
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Misgendering is the act of referring to someone in a way that does not
reflect their gender identity. Translation systems, including foundation models
capable of translation, can produce errors that result in misgendering harms.
To measure the extent of such potential harms when translating into and out of
English, we introduce a dataset, MiTTenS, covering 26 languages from a variety
of language families and scripts, including several traditionally
underpresented in digital resources. The dataset is constructed with
handcrafted passages that target known failure patterns, longer synthetically
generated passages, and natural passages sourced from multiple domains. We
demonstrate the usefulness of the dataset by evaluating both dedicated neural
machine translation systems and foundation models, and show that all systems
exhibit errors resulting in misgendering harms, even in high resource
languages.
Related papers
- Beyond Binary Gender: Evaluating Gender-Inclusive Machine Translation with Ambiguous Attitude Words [85.48043537327258]
Existing machine translation gender bias evaluations are primarily focused on male and female genders.
This study presents a benchmark AmbGIMT (Gender-Inclusive Machine Translation with Ambiguous attitude words)
We propose a novel process to evaluate gender bias based on the Emotional Attitude Score (EAS), which is used to quantify ambiguous attitude words.
arXiv Detail & Related papers (2024-07-23T08:13:51Z) - Multilingual Text-to-Image Generation Magnifies Gender Stereotypes and Prompt Engineering May Not Help You [64.74707085021858]
We show that multilingual models suffer from significant gender biases just as monolingual models do.
We propose a novel benchmark, MAGBIG, intended to foster research on gender bias in multilingual models.
Our results show that not only do models exhibit strong gender biases but they also behave differently across languages.
arXiv Detail & Related papers (2024-01-29T12:02:28Z) - Reducing Gender Bias in Machine Translation through Counterfactual Data
Generation [0.0]
We show that gender bias can be significantly mitigated, albeit at the expense of translation quality due to catastrophic forgetting.
We also propose a novel domain-adaptation technique that leverages in-domain data created with the counterfactual data generation techniques.
The relevant code will be available at Github.
arXiv Detail & Related papers (2023-11-27T23:03:01Z) - Target-Agnostic Gender-Aware Contrastive Learning for Mitigating Bias in
Multilingual Machine Translation [28.471506840241602]
Gender bias is a significant issue in machine translation, leading to ongoing research efforts in developing bias mitigation techniques.
We propose a bias mitigation method based on a novel approach.
Gender-Aware Contrastive Learning, GACL, encodes contextual gender information into the representations of non-explicit gender words.
arXiv Detail & Related papers (2023-05-23T12:53:39Z) - GATE: A Challenge Set for Gender-Ambiguous Translation Examples [0.31498833540989407]
When source gender is ambiguous, machine translation models typically default to stereotypical gender roles, perpetuating harmful bias.
Recent work has led to the development of "gender rewriters" that generate alternative gender translations on such ambiguous inputs, but such systems are plagued by poor linguistic coverage.
We present and release GATE, a linguistically diverse corpus of gender-ambiguous source sentences along with multiple alternative target language translations.
arXiv Detail & Related papers (2023-03-07T15:23:38Z) - Evaluating Gender Bias in Hindi-English Machine Translation [0.1503974529275767]
We implement a modified version of the TGBI metric based on the grammatical considerations for Hindi.
We compare and contrast the resulting bias measurements across multiple metrics for pre-trained embeddings and the ones learned by our machine translation model.
arXiv Detail & Related papers (2021-06-16T10:35:51Z) - Investigating Failures of Automatic Translation in the Case of
Unambiguous Gender [13.58884863186619]
Transformer based models are the modern work horses for neural machine translation (NMT)
We observe a systemic and rudimentary class of errors made by transformer based models with regards to translating from a language that doesn't mark gender on nouns into others that do.
We release an evaluation scheme and dataset for measuring the ability of transformer based NMT models to translate gender correctly.
arXiv Detail & Related papers (2021-04-16T00:57:36Z) - Curious Case of Language Generation Evaluation Metrics: A Cautionary
Tale [52.663117551150954]
A few popular metrics remain as the de facto metrics to evaluate tasks such as image captioning and machine translation.
This is partly due to ease of use, and partly because researchers expect to see them and know how to interpret them.
In this paper, we urge the community for more careful consideration of how they automatically evaluate their models.
arXiv Detail & Related papers (2020-10-26T13:57:20Z) - Mitigating Gender Bias in Captioning Systems [56.25457065032423]
Most captioning models learn gender bias, leading to high gender prediction errors, especially for women.
We propose a new Guided Attention Image Captioning model (GAIC) which provides self-guidance on visual attention to encourage the model to capture correct gender visual evidence.
arXiv Detail & Related papers (2020-06-15T12:16:19Z) - Gender Bias in Multilingual Embeddings and Cross-Lingual Transfer [101.58431011820755]
We study gender bias in multilingual embeddings and how it affects transfer learning for NLP applications.
We create a multilingual dataset for bias analysis and propose several ways for quantifying bias in multilingual representations.
arXiv Detail & Related papers (2020-05-02T04:34:37Z) - Multi-Dimensional Gender Bias Classification [67.65551687580552]
Machine learning models can inadvertently learn socially undesirable patterns when training on gender biased text.
We propose a general framework that decomposes gender bias in text along several pragmatic and semantic dimensions.
Using this fine-grained framework, we automatically annotate eight large scale datasets with gender information.
arXiv Detail & Related papers (2020-05-01T21:23:20Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.