Bias in the Ear of the Listener: Assessing Sensitivity in Audio Language Models Across Linguistic, Demographic, and Positional Variations

• URL: http://arxiv.org/abs/2602.01030v1
• Date: Sun, 01 Feb 2026 05:34:34 GMT
• Title: Bias in the Ear of the Listener: Assessing Sensitivity in Audio Language Models Across Linguistic, Demographic, and Positional Variations
• Authors: Sheng-Lun Wei, Yu-Ling Liao, Yen-Hua Chang, Hen-Hsen Huang, Hsin-Hsi Chen,
• Abstract summary: We construct and release the BiasInEar dataset, a speech-augmented benchmark based on Global MMLU Lite.<n>We evaluate nine representative models under linguistic (language and accent), demographic (gender), and structural (option order) perturbations.<n>Our findings reveal that MLLMs are relatively robust to demographic factors but highly sensitive to language and option order, suggesting that speech can amplify existing structural biases.
• Score: 21.050704978484784
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This work presents the first systematic investigation of speech bias in multilingual MLLMs. We construct and release the BiasInEar dataset, a speech-augmented benchmark based on Global MMLU Lite, spanning English, Chinese, and Korean, balanced by gender and accent, and totaling 70.8 hours ($\approx$4,249 minutes) of speech with 11,200 questions. Using four complementary metrics (accuracy, entropy, APES, and Fleiss' $κ$), we evaluate nine representative models under linguistic (language and accent), demographic (gender), and structural (option order) perturbations. Our findings reveal that MLLMs are relatively robust to demographic factors but highly sensitive to language and option order, suggesting that speech can amplify existing structural biases. Moreover, architectural design and reasoning strategy substantially affect robustness across languages. Overall, this study establishes a unified framework for assessing fairness and robustness in speech-integrated LLMs, bridging the gap between text- and speech-based evaluation. The resources can be found at https://github.com/ntunlplab/BiasInEar.

Related papers

• Does Language Model Understand Language? [1.0450509067356148]
  Despite advances in natural language generation and understanding, LM still struggle with fine grained linguistic phenomena.<n>In this study, we conduct a evaluation of SOTA language models across challenging contexts in both English and Bengali.<n>Our findings highlight Compound-Beta as the most balanced model, consistently achieving high correlations and low MAEs across diverse language conditions.
  arXiv  Detail & Related papers  (2025-09-15T21:09:09Z)
• Language-agnostic, automated assessment of listeners' speech recall using large language models [0.0]
  This research leverages modern large language models (LLMs) in native English speakers and native speakers of 10 other languages.<n>Participants listened to and freely recalled short stories (in quiet/clear and in babble noise) in their native language.<n>LLMs prompt engineering with semantic similarity analyses to score speech recall revealed sensitivity to known effects of temporal order, primacy/recency, and background noise.
  arXiv  Detail & Related papers  (2025-03-02T22:28:41Z)
• Language Barriers: Evaluating Cross-Lingual Performance of CNN and Transformer Architectures for Speech Quality Estimation [9.286959744769792]
  Cross-lingual generalization of objective speech quality models is a major challenge.<n>Models trained primarily on English data may struggle to generalize to languages with different phonetic, tonal, and prosodic characteristics.<n>This study investigates the cross-lingual performance of two speech quality models: NISQA, a CNN-based model, and a Transformer-based Audio Spectrogram Transformer (AST) model.
  arXiv  Detail & Related papers  (2025-02-18T16:22:43Z)
• Audio Large Language Models Can Be Descriptive Speech Quality Evaluators [46.765203628127345]
  We introduce the first natural language-based speech evaluation corpus, generated from authentic human ratings.<n>This corpus offers detailed analysis across multiple dimensions and identifies causes of quality degradation.<n>We propose an alignment approach with LLM distillation (ALLD) to guide the audio LLM in extracting relevant information from raw speech.
  arXiv  Detail & Related papers  (2025-01-27T22:47:51Z)
• Assessing Dialect Fairness and Robustness of Large Language Models in Reasoning Tasks [68.33068005789116]
  We introduce ReDial, a benchmark containing 1.2K+ parallel query pairs in Standardized English and AAVE.<n>We evaluate widely used models, including GPT, Claude, Llama, Mistral, and the Phi model families.<n>Our work establishes a systematic and objective framework for analyzing LLM bias in dialectal queries.
  arXiv  Detail & Related papers  (2024-10-14T18:44:23Z)
• Listen and Speak Fairly: A Study on Semantic Gender Bias in Speech Integrated Large Language Models [38.64792118903994]
  We evaluate gender bias in SILLMs across four semantic-related tasks. Our analysis reveals that bias levels are language-dependent and vary with different evaluation methods.
  arXiv  Detail & Related papers  (2024-07-09T15:35:43Z)
• An Initial Investigation of Language Adaptation for TTS Systems under Low-resource Scenarios [76.11409260727459]
  This paper explores the language adaptation capability of ZMM-TTS, a recent SSL-based multilingual TTS system. We demonstrate that the similarity in phonetics between the pre-training and target languages, as well as the language category, affects the target language's adaptation performance.
  arXiv  Detail & Related papers  (2024-06-13T08:16:52Z)
• Quantifying the Dialect Gap and its Correlates Across Languages [69.18461982439031]
  This work will lay the foundation for furthering the field of dialectal NLP by laying out evident disparities and identifying possible pathways for addressing them through mindful data collection.
  arXiv  Detail & Related papers  (2023-10-23T17:42:01Z)
• Disco-Bench: A Discourse-Aware Evaluation Benchmark for Language Modelling [70.23876429382969]
  We propose a benchmark that can evaluate intra-sentence discourse properties across a diverse set of NLP tasks. Disco-Bench consists of 9 document-level testsets in the literature domain, which contain rich discourse phenomena. For linguistic analysis, we also design a diagnostic test suite that can examine whether the target models learn discourse knowledge.
  arXiv  Detail & Related papers  (2023-07-16T15:18:25Z)
• Towards Language Modelling in the Speech Domain Using Sub-word Linguistic Units [56.52704348773307]
  We propose a novel LSTM-based generative speech LM based on linguistic units including syllables and phonemes. With a limited dataset, orders of magnitude smaller than that required by contemporary generative models, our model closely approximates babbling speech. We show the effect of training with auxiliary text LMs, multitask learning objectives, and auxiliary articulatory features.
  arXiv  Detail & Related papers  (2021-10-31T22:48:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.