Measuring Gender Bias in Job Title Matching for Grammatical Gender Languages

• URL: http://arxiv.org/abs/2509.13803v1
• Date: Wed, 17 Sep 2025 08:17:28 GMT
• Title: Measuring Gender Bias in Job Title Matching for Grammatical Gender Languages
• Authors: Laura García-Sardiña, Hermenegildo Fabregat, Daniel Deniz, Rabih Zbib,
• Abstract summary: This work sets the ground for studying how explicit grammatical gender assignment in job titles can affect the results of automatic job ranking systems.<n>We propose the usage of metrics for ranking comparison controlling for gender to evaluate gender bias in job title ranking systems.
• Score: 0.07666382766590014
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This work sets the ground for studying how explicit grammatical gender assignment in job titles can affect the results of automatic job ranking systems. We propose the usage of metrics for ranking comparison controlling for gender to evaluate gender bias in job title ranking systems, in particular RBO (Rank-Biased Overlap). We generate and share test sets for a job title matching task in four grammatical gender languages, including occupations in masculine and feminine form and annotated by gender and matching relevance. We use the new test sets and the proposed methodology to evaluate the gender bias of several out-of-the-box multilingual models to set as baselines, showing that all of them exhibit varying degrees of gender bias.

Related papers

• GAMBIT+: A Challenge Set for Evaluating Gender Bias in Machine Translation Quality Estimation Metrics [18.766033854102663]
  Gender bias in machine translation (MT) systems has been extensively documented, but bias in automatic quality estimation (QE) metrics remains comparatively underexplored.<n>Existing studies suggest that QE metrics can also exhibit gender bias, yet most analyses are limited by small datasets, narrow occupational coverage, and restricted language variety.<n>We introduce a large-scale challenge set specifically designed to probe the behavior of QE metrics when evaluating translations containing gender-ambiguous occupational terms.
  arXiv  Detail & Related papers  (2025-10-08T10:09:03Z)
• Exploring Gender Bias Beyond Occupational Titles [1.2123876307427102]
  We introduce a novel dataset, GenderLexicon, and a framework that can estimate contextual bias and its related gender bias.<n>Our model can interpret the bias with a score and thus improve the explainability of gender bias.
  arXiv  Detail & Related papers  (2025-07-03T14:42:03Z)
• Assumed Identities: Quantifying Gender Bias in Machine Translation of Gender-Ambiguous Occupational Terms [2.5764960393034615]
  We introduce GRAPE, a probability-based metric designed to evaluate gender bias.<n>We present GAMBIT-MT, a benchmarking dataset in English with gender-ambiguous occupational terms.
  arXiv  Detail & Related papers  (2025-03-06T12:16:14Z)
• Identifying Gender Stereotypes and Biases in Automated Translation from English to Italian using Similarity Networks [0.25049267048783647]
  This paper is a collaborative effort between Linguistics, Law, and Computer Science to evaluate stereotypes and biases in automated translation systems.<n>We advocate gender-neutral translation as a means to promote gender inclusion and improve the objectivity of machine translation.
  arXiv  Detail & Related papers  (2025-02-17T09:55:32Z)
• Beyond Binary Gender: Evaluating Gender-Inclusive Machine Translation with Ambiguous Attitude Words [85.48043537327258]
  Existing machine translation gender bias evaluations are primarily focused on male and female genders. This study presents a benchmark AmbGIMT (Gender-Inclusive Machine Translation with Ambiguous attitude words) We propose a novel process to evaluate gender bias based on the Emotional Attitude Score (EAS), which is used to quantify ambiguous attitude words.
  arXiv  Detail & Related papers  (2024-07-23T08:13:51Z)
• GenderBias-\emph{VL}: Benchmarking Gender Bias in Vision Language Models via Counterfactual Probing [72.0343083866144]
  This paper introduces the GenderBias-emphVL benchmark to evaluate occupation-related gender bias in Large Vision-Language Models. Using our benchmark, we extensively evaluate 15 commonly used open-source LVLMs and state-of-the-art commercial APIs. Our findings reveal widespread gender biases in existing LVLMs.
  arXiv  Detail & Related papers  (2024-06-30T05:55:15Z)
• VisoGender: A dataset for benchmarking gender bias in image-text pronoun resolution [80.57383975987676]
  VisoGender is a novel dataset for benchmarking gender bias in vision-language models. We focus on occupation-related biases within a hegemonic system of binary gender, inspired by Winograd and Winogender schemas. We benchmark several state-of-the-art vision-language models and find that they demonstrate bias in resolving binary gender in complex scenes.
  arXiv  Detail & Related papers  (2023-06-21T17:59:51Z)
• Social Biases in Automatic Evaluation Metrics for NLG [53.76118154594404]
  We propose an evaluation method based on Word Embeddings Association Test (WEAT) and Sentence Embeddings Association Test (SEAT) to quantify social biases in evaluation metrics. We construct gender-swapped meta-evaluation datasets to explore the potential impact of gender bias in image caption and text summarization tasks.
  arXiv  Detail & Related papers  (2022-10-17T08:55:26Z)
• Measuring Gender Bias in Word Embeddings of Gendered Languages Requires Disentangling Grammatical Gender Signals [3.0349733976070015]
  We demonstrate that word embeddings learn the association between a noun and its grammatical gender in grammatically gendered languages. We show that disentangling grammatical gender signals from word embeddings may lead to improvement in semantic machine learning tasks.
  arXiv  Detail & Related papers  (2022-06-03T17:11:00Z)
• They, Them, Theirs: Rewriting with Gender-Neutral English [56.14842450974887]
  We perform a case study on the singular they, a common way to promote gender inclusion in English. We show how a model can be trained to produce gender-neutral English with 1% word error rate with no human-labeled data.
  arXiv  Detail & Related papers  (2021-02-12T21:47:48Z)
• Multi-Dimensional Gender Bias Classification [67.65551687580552]
  Machine learning models can inadvertently learn socially undesirable patterns when training on gender biased text. We propose a general framework that decomposes gender bias in text along several pragmatic and semantic dimensions. Using this fine-grained framework, we automatically annotate eight large scale datasets with gender information.
  arXiv  Detail & Related papers  (2020-05-01T21:23:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.