Measuring Fairness in Large Audio Language Models via Semantic-Aware Bias Estimation

• URL: http://arxiv.org/abs/2608.13624v1
• Date: Thu, 13 Aug 2026 03:24:49 GMT
• Title: Measuring Fairness in Large Audio Language Models via Semantic-Aware Bias Estimation
• Abstract summary: Large Audio Language Models (LALMs) have seen increasing use for audio understanding tasks such as speech recognition and audio question answering.<n>We propose a semantic-aware mixed-effects regression framework for fairness evaluation in LALMs.
• Score: 5.67379336475756
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Audio Language Models (LALMs) have seen increasing use for audio understanding tasks such as speech recognition and audio question answering, raising concerns about fairness across demographic subgroups. Fairness evaluation in spoken-input settings is challenging due to confounding factors, including semantic variation in spoken content and speaker-specific characteristics. Ignoring these factors can result in misleading conclusions about model bias. We propose a semantic-aware mixed-effects regression framework for fairness evaluation in LALMs that explicitly accounts for these confounders. Our approach incorporates sentence-level semantic embeddings of reference text as covariates and models speaker identity as a random effect. Notably, semantic representations are extracted from the same LALM under evaluation, enabling semantic control over variation as perceived by the model itself. Experiments on simulated data and real-world benchmarks demonstrate that the proposed approach substantially reduces spurious fairness findings and yields more robust and interpretable estimates of subgroup performance differences.

Related papers

• Is my model perplexed for the right reason? Contrasting LLMs' Benchmark Behavior with Token-Level Perplexity [6.285055089721361]
  We introduce a principled interpretability framework based on token-level perplexity to test whether models rely on linguistically relevant cues.<n>Our method enables precise, hypothesis-driven analysis without relying on unstable feature-attribution techniques.
  arXiv  Detail & Related papers  (2026-03-31T08:00:55Z)
• Membership Inference Attacks against Large Audio Language Models [50.84901010528239]
  We present the first systematic Membership Inference Attack (MIA) evaluation of Large Audio Language Models (LALMs)<n>Using a multi-modal blind baseline based on textual, spectral, and prosodic features, we demonstrate that common speech datasets exhibit near-perfect train/test separability.<n>These findings establish a principled standard for auditing LALMs beyond spurious correlations.
  arXiv  Detail & Related papers  (2026-03-30T12:45:28Z)
• Evaluation of Audio Language Models for Fairness, Safety, and Security [10.580283172294733]
  We introduce a structural taxonomy (system-level and representational) of audio large language models (ALLMs)<n>We propose a unified evaluation framework that assesses semantic invariance under paralinguistic variation, refusal and toxicity behavior under unsafe prompts, and robustness to adversarial audio perturbations.<n>Our findings demonstrate that FSS behavior is tightly coupled to how acoustic information is integrated into semantic reasoning, underscoring the need for structure-aware evaluation of audio language models.
  arXiv  Detail & Related papers  (2026-02-25T21:22:28Z)
• AQAScore: Evaluating Semantic Alignment in Text-to-Audio Generation via Audio Question Answering [97.52852990265136]
  We introduce AQAScore, a backbone-agnostic evaluation framework that leverages the reasoning capabilities of audio-aware large language models.<n>We evaluate AQAScore across multiple benchmarks, including human-rated relevance, pairwise comparison, and compositional reasoning tasks.
  arXiv  Detail & Related papers  (2026-01-21T07:35:36Z)
• On the Fallacy of Global Token Perplexity in Spoken Language Model Evaluation [88.77441715819366]
  Generative spoken language models pretrained on large-scale raw audio can continue a speech prompt with appropriate content.<n>We propose a variety of likelihood- and generative-based evaluation methods that serve in place of naive global token perplexity.
  arXiv  Detail & Related papers  (2026-01-09T22:01:56Z)
• Quantifying Fairness in LLMs Beyond Tokens: A Semantic and Statistical Perspective [24.54292750583169]
  Large Language Models (LLMs) often generate responses with inherent biases, undermining their reliability in real-world applications.<n>We propose FiSCo (Fine-grained Semantic Comparison), a novel statistical framework to evaluate group-level fairness in LLMs.<n>We decompose model outputs into semantically distinct claims and apply statistical hypothesis testing to compare inter- and intra-group similarities.
  arXiv  Detail & Related papers  (2025-06-23T18:31:22Z)
• CO-VADA: A Confidence-Oriented Voice Augmentation Debiasing Approach for Fair Speech Emotion Recognition [49.27067541740956]
  We present CO-VADA, a Confidence-Oriented Voice Augmentation Debiasing Approach that mitigates bias without modifying model architecture or relying on demographic information.<n>CO-VADA identifies training samples that reflect bias patterns present in the training data and then applies voice conversion to alter irrelevant attributes and generate samples.<n>Our framework is compatible with various SER models and voice conversion tools, making it a scalable and practical solution for improving fairness in SER systems.
  arXiv  Detail & Related papers  (2025-06-06T13:25:56Z)
• $C^2$AV-TSE: Context and Confidence-aware Audio Visual Target Speaker Extraction [80.57232374640911]
  We propose a model-agnostic strategy called the Mask-And-Recover (MAR)<n>MAR integrates both inter- and intra-modality contextual correlations to enable global inference within extraction modules.<n>To better target challenging parts within each sample, we introduce a Fine-grained Confidence Score (FCS) model.
  arXiv  Detail & Related papers  (2025-04-01T13:01:30Z)
• Estimating Commonsense Plausibility through Semantic Shifts [66.06254418551737]
  We propose ComPaSS, a novel discriminative framework that quantifies commonsense plausibility by measuring semantic shifts.<n> Evaluations on two types of fine-grained commonsense plausibility estimation tasks show that ComPaSS consistently outperforms baselines.
  arXiv  Detail & Related papers  (2025-02-19T06:31:06Z)
• All Should Be Equal in the Eyes of Language Models: Counterfactually Aware Fair Text Generation [16.016546693767403]
  We propose a framework that dynamically compares the model understanding of diverse demographics to generate more equitable sentences. CAFIE produces fairer text and strikes the best balance between fairness and language modeling capability.
  arXiv  Detail & Related papers  (2023-11-09T15:39:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.