Demographic Biases and Gaps in the Perception of Sexism in Large Language Models

• URL: http://arxiv.org/abs/2508.18245v1
• Date: Mon, 25 Aug 2025 17:36:58 GMT
• Title: Demographic Biases and Gaps in the Perception of Sexism in Large Language Models
• Authors: Judith Tavarez-Rodríguez, Fernando Sánchez-Vega, A. Pastor López-Monroy,
• Abstract summary: We explore the capabilities of different Large Language Models to detect sexism in social media text.<n>We analyze the demographic biases present in the models and conduct a statistical analysis.<n>Our results show that, while LLMs can to some extent detect sexism when considering the overall opinion of populations, they do not accurately replicate the diversity of perceptions among different demographic groups.
• Score: 43.77504559722899
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The use of Large Language Models (LLMs) has proven to be a tool that could help in the automatic detection of sexism. Previous studies have shown that these models contain biases that do not accurately reflect reality, especially for minority groups. Despite various efforts to improve the detection of sexist content, this task remains a significant challenge due to its subjective nature and the biases present in automated models. We explore the capabilities of different LLMs to detect sexism in social media text using the EXIST 2024 tweet dataset. It includes annotations from six distinct profiles for each tweet, allowing us to evaluate to what extent LLMs can mimic these groups' perceptions in sexism detection. Additionally, we analyze the demographic biases present in the models and conduct a statistical analysis to identify which demographic characteristics (age, gender) contribute most effectively to this task. Our results show that, while LLMs can to some extent detect sexism when considering the overall opinion of populations, they do not accurately replicate the diversity of perceptions among different demographic groups. This highlights the need for better-calibrated models that account for the diversity of perspectives across different populations.

Related papers

• Measuring Social Bias in Vision-Language Models with Face-Only Counterfactuals from Real Photos [79.03150233804458]
  Real-world images entangle race and gender with correlated factors such as background and clothing, obscuring attribution.<n>We propose a textbfface-only counterfactual evaluation paradigm<n>We generate counterfactual variants by editing only facial attributes related to race and gender, keeping all other visual factors fixed.
  arXiv  Detail & Related papers  (2026-01-11T14:35:06Z)
• The Effects of Demographic Instructions on LLM Personas [14.283869154967835]
  Social media platforms must filter sexist content in compliance with governmental regulations.<n>Current machine learning approaches can reliably detect sexism based on standardized definitions.<n>We adopt a perspectivist approach, retaining diverse annotations rather than enforcing gold-standard labels.
  arXiv  Detail & Related papers  (2025-05-17T02:49:15Z)
• More Women, Same Stereotypes: Unpacking the Gender Bias Paradox in Large Language Models [3.154053412440065]
  Large Language Models (LLMs) have revolutionized natural language processing, yet concerns persist regarding their tendency to reflect or amplify social biases.<n>This study introduces a novel evaluation framework to uncover gender biases in LLMs.<n>A systematic analysis of ten prominent LLMs shows a consistent pattern of overrepresenting female characters across occupations.
  arXiv  Detail & Related papers  (2025-03-20T07:15:45Z)
• Small Changes, Large Consequences: Analyzing the Allocational Fairness of LLMs in Hiring Contexts [19.20592062296075]
  Large language models (LLMs) are increasingly being deployed in high-stakes applications like hiring.<n>This work examines the allocational fairness of LLM-based hiring systems through two tasks that reflect actual HR usage.
  arXiv  Detail & Related papers  (2025-01-08T07:28:10Z)
• The Root Shapes the Fruit: On the Persistence of Gender-Exclusive Harms in Aligned Language Models [91.86718720024825]
  We center transgender, nonbinary, and other gender-diverse identities to investigate how alignment procedures interact with pre-existing gender-diverse bias.<n>Our findings reveal that DPO-aligned models are particularly sensitive to supervised finetuning.<n>We conclude with recommendations tailored to DPO and broader alignment practices.
  arXiv  Detail & Related papers  (2024-11-06T06:50:50Z)
• Revealing and Reducing Gender Biases in Vision and Language Assistants (VLAs) [82.57490175399693]
  We study gender bias in 22 popular image-to-text vision-language assistants (VLAs)<n>Our results show that VLAs replicate human biases likely present in the data, such as real-world occupational imbalances.<n>To eliminate the gender bias in these models, we find that fine-tuning-based debiasing methods achieve the best trade-off between debiasing and retaining performance.
  arXiv  Detail & Related papers  (2024-10-25T05:59:44Z)
• GenderCARE: A Comprehensive Framework for Assessing and Reducing Gender Bias in Large Language Models [73.23743278545321]
  Large language models (LLMs) have exhibited remarkable capabilities in natural language generation, but have also been observed to magnify societal biases.<n>GenderCARE is a comprehensive framework that encompasses innovative Criteria, bias Assessment, Reduction techniques, and Evaluation metrics.
  arXiv  Detail & Related papers  (2024-08-22T15:35:46Z)
• GenderBias-\emph{VL}: Benchmarking Gender Bias in Vision Language Models via Counterfactual Probing [72.0343083866144]
  This paper introduces the GenderBias-emphVL benchmark to evaluate occupation-related gender bias in Large Vision-Language Models. Using our benchmark, we extensively evaluate 15 commonly used open-source LVLMs and state-of-the-art commercial APIs. Our findings reveal widespread gender biases in existing LVLMs.
  arXiv  Detail & Related papers  (2024-06-30T05:55:15Z)
• Sociodemographic Prompting is Not Yet an Effective Approach for Simulating Subjective Judgments with LLMs [13.744746481528711]
  Large Language Models (LLMs) are widely used to simulate human responses across diverse contexts.<n>We evaluate nine popular LLMs on their ability to understand demographic differences in two subjective judgment tasks: politeness and offensiveness.<n>We find that in zero-shot settings, most models' predictions for both tasks align more closely with labels from White participants than those from Asian or Black participants.
  arXiv  Detail & Related papers  (2023-11-16T10:02:24Z)
• Probing Explicit and Implicit Gender Bias through LLM Conditional Text Generation [64.79319733514266]
  Large Language Models (LLMs) can generate biased and toxic responses. We propose a conditional text generation mechanism without the need for predefined gender phrases and stereotypes.
  arXiv  Detail & Related papers  (2023-11-01T05:31:46Z)
• "Call me sexist, but...": Revisiting Sexism Detection Using Psychological Scales and Adversarial Samples [2.029924828197095]
  We outline the different dimensions of sexism by grounding them in their implementation in psychological scales. From the scales, we derive a codebook for sexism in social media, which we use to annotate existing and novel datasets. Results indicate that current machine learning models pick up on a very narrow set of linguistic markers of sexism and do not generalize well to out-of-domain examples.
  arXiv  Detail & Related papers  (2020-04-27T13:07:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.