Demographic and Linguistic Bias Evaluation in Omnimodal Language Models

• URL: http://arxiv.org/abs/2604.10014v1
• Date: Sat, 11 Apr 2026 03:58:02 GMT
• Title: Demographic and Linguistic Bias Evaluation in Omnimodal Language Models
• Abstract summary: Four omnimodal models are evaluated on tasks that include demographic attribute estimation, identity verification, activity recognition, multilingual speech transcription, and language identification.<n>Results show that image and video understanding tasks generally exhibit better performance with smaller demographic disparities.<n>In contrast, audio understanding tasks exhibit significantly lower performance and substantial bias, including large accuracy differences across age groups, genders, and languages, and frequent prediction collapse toward narrow categories.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper provides a comprehensive evaluation of demographic and linguistic biases in omnimodal language models that process text, images, audio, and video within a single framework. Although these models are being widely deployed, their performance across different demographic groups and modalities is not well studied. Four omnimodal models are evaluated on tasks that include demographic attribute estimation, identity verification, activity recognition, multilingual speech transcription, and language identification. Accuracy differences are measured across age, gender, skin tone, language, and country of origin. The results show that image and video understanding tasks generally exhibit better performance with smaller demographic disparities. In contrast, audio understanding tasks exhibit significantly lower performance and substantial bias, including large accuracy differences across age groups, genders, and languages, and frequent prediction collapse toward narrow categories. These findings highlight the importance of evaluating fairness across all supported modalities as omnimodal language models are increasingly used in real-world applications.

Related papers

• FairForensics: Seeing Expressions and Parsing Demographics via Vision-Language Modeling for Generalizable Fair Deepfake Detection [60.2868121662026]
  Existing fairness-enhanced detectors often suffer from suboptimal generalization to unseen manipulations and fairness across demographic groups.<n>We construct a novel demographically balanced FDD benchmark to train and evaluate the fairness of detectors under both balanced and imbalanced population scenarios.<n>We also introduce a novel expression and demographic perceptual vision-language model, termed FairForensics, for generalizable fair deepfake detection.
  arXiv  Detail & Related papers  (2026-08-03T03:51:14Z)
• Your Multimodal Speech Model Says I Have a Face for Radio [10.918147393269352]
  We show the first bias evaluation of multimodal speech recognition.<n>We find large quality-of-service differences across mWhisper-Flamingo and Gemini models, with drops of up to 4.05 word error rate points.<n>Our findings point to a priority for developers to evaluate, fix, and communicate such limitations.
  arXiv  Detail & Related papers  (2026-05-28T18:42:38Z)
• On the Fallacy of Global Token Perplexity in Spoken Language Model Evaluation [88.77441715819366]
  Generative spoken language models pretrained on large-scale raw audio can continue a speech prompt with appropriate content.<n>We propose a variety of likelihood- and generative-based evaluation methods that serve in place of naive global token perplexity.
  arXiv  Detail & Related papers  (2026-01-09T22:01:56Z)
• Experiential Semantic Information and Brain Alignment: Are Multimodal Models Better than Language Models? [5.412335160966597]
  A common assumption in Computational Linguistics is that text representations learnt by multimodal models are richer and more human-like than those by language-only models.<n>We compare word representations from contrastive multimodal models vs. language-only ones in the extent to which they capture information.<n>Our results indicate that, surprisingly, language-only models are superior to multimodal ones in both respects.
  arXiv  Detail & Related papers  (2025-04-01T16:28:38Z)
• Demographic Attributes Prediction from Speech Using WavLM Embeddings [25.00298717665857]
  This paper introduces a general classifier based on WavLM features, to infer demographic characteristics, such as age, gender, native language, education, and country, from speech.<n>The proposed framework identifies key acoustic and linguistic fea-tures associated with demographic attributes, achieving a Mean Absolute Error (MAE) of 4.94 for age prediction and over 99.81% accuracy for gender classification.
  arXiv  Detail & Related papers  (2025-02-17T16:43:47Z)
• Data Matters Most: Auditing Social Bias in Contrastive Vision Language Models [28.944990804599893]
  We disentangle three design factors -- model size, training-data scale, and training-data source -- by comparing CLIP and OpenCLIP.<n>We also evaluate three post-hoc, test-time debiasing strategies -- Bias Prompts, Prompt Array, and SANER.
  arXiv  Detail & Related papers  (2025-01-22T21:08:30Z)
• Spoken Stereoset: On Evaluating Social Bias Toward Speaker in Speech Large Language Models [50.40276881893513]
  This study introduces Spoken Stereoset, a dataset specifically designed to evaluate social biases in Speech Large Language Models (SLLMs) By examining how different models respond to speech from diverse demographic groups, we aim to identify these biases. The findings indicate that while most models show minimal bias, some still exhibit slightly stereotypical or anti-stereotypical tendencies.
  arXiv  Detail & Related papers  (2024-08-14T16:55:06Z)
• Sonos Voice Control Bias Assessment Dataset: A Methodology for Demographic Bias Assessment in Voice Assistants [10.227469020901232]
  This paper introduces the Sonos Voice Control Bias Assessment dataset. 1,038 speakers, 166 hours, 170k audio samples, with 9,040 unique labelled transcripts. Results show statistically significant differences in performance across age, dialectal region and ethnicity.
  arXiv  Detail & Related papers  (2024-05-14T12:53:32Z)
• Multilingual Text-to-Image Generation Magnifies Gender Stereotypes and Prompt Engineering May Not Help You [64.74707085021858]
  We show that multilingual models suffer from significant gender biases just as monolingual models do.<n>We propose a novel benchmark, MAGBIG, intended to foster research on gender bias in multilingual models.<n>Our results show that not only do models exhibit strong gender biases but they also behave differently across languages.
  arXiv  Detail & Related papers  (2024-01-29T12:02:28Z)
• Quantifying the Dialect Gap and its Correlates Across Languages [69.18461982439031]
  This work will lay the foundation for furthering the field of dialectal NLP by laying out evident disparities and identifying possible pathways for addressing them through mindful data collection.
  arXiv  Detail & Related papers  (2023-10-23T17:42:01Z)
• Measuring Geographic Performance Disparities of Offensive Language Classifiers [12.545108947857802]
  We ask two questions: Does language, dialect, and topical content vary across geographical regions?'' and If there are differences across the regions, do they impact model performance?'' We find that current models do not generalize across locations. Likewise, we show that while offensive language models produce false positives on African American English, model performance is not correlated with each city's minority population proportions.
  arXiv  Detail & Related papers  (2022-09-15T15:08:18Z)
• Analyzing the Limits of Self-Supervision in Handling Bias in Language [52.26068057260399]
  We evaluate how well language models capture the semantics of four tasks for bias: diagnosis, identification, extraction and rephrasing. Our analyses indicate that language models are capable of performing these tasks to widely varying degrees across different bias dimensions, such as gender and political affiliation.
  arXiv  Detail & Related papers  (2021-12-16T05:36:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.