Do LLM Decoders Listen Fairly? Benchmarking How Language Model Priors Shape Bias in Speech Recognition

• URL: http://arxiv.org/abs/2604.21276v1
• Date: Thu, 23 Apr 2026 04:40:58 GMT
• Title: Do LLM Decoders Listen Fairly? Benchmarking How Language Model Priors Shape Bias in Speech Recognition
• Abstract summary: Pretrained large language models replace task-specific decoders in speech recognition.<n>Do their text-derived priors make recognition fairer or more biased across demographic groups?<n>We evaluate nine models spanning three architectural generations on about 43,000 utterances across five demographic axes.
• Score: 9.131439623985028
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As pretrained large language models replace task-specific decoders in speech recognition, a critical question arises: do their text-derived priors make recognition fairer or more biased across demographic groups? We evaluate nine models spanning three architectural generations (CTC with no language model, encoder-decoder with an implicit LM, and LLM-based with an explicit pretrained decoder) on about 43,000 utterances across five demographic axes (ethnicity, accent, gender, age, first language) using Common Voice 24 and Meta's Fair-Speech, a controlled-prompt dataset that eliminates vocabulary confounds. On clean audio, three findings challenge assumptions: LLM decoders do not amplify racial bias (Granite-8B has the best ethnicity fairness, max/min WER = 2.28); Whisper exhibits pathological hallucination on Indian-accented speech with a non-monotonic insertion-rate spike to 9.62% at large-v3; and audio compression predicts accent fairness more than LLM scale. We then stress-test these findings under 12 acoustic degradation conditions (noise, reverberation, silence injection, chunk masking) across both datasets, totaling 216 inference runs. Severe degradation paradoxically compresses fairness gaps as all groups converge to high WER, but silence injection amplifies Whisper's accent bias up to 4.64x by triggering demographic-selective hallucination. Under masking, Whisper enters catastrophic repetition loops (86% of 51,797 insertions) while explicit-LLM decoders produce 38x fewer insertions with near-zero repetition; high-compression audio encoding (Q-former) reintroduces repetition pathology even in LLM decoders. These results suggest that audio encoder design, not LLM scaling, is the primary lever for equitable and robust speech recognition.

Related papers

• Heard but Not Heeded: Paralinguistic Information Encoding and Loss in Audio-Language Models [58.79935033105039]
  We present a mechanistic analysis of paralinguistic information in four open source models, Whisper-large-v2, Qwen2-Audio-7B Instruct, Qwen2.5- Omni-7B, and Chroma-4B.<n>We combine centered kernel alignment, linear probing with leave one speaker out evaluation, open ended tone prediction, and a content prosody leakage metric to trace how style information moves from the audio encoder to the final output.
  arXiv  Detail & Related papers  (2026-09-01T05:03:33Z)
• Audio-Native Speech Recognition with a Frozen Discrete-Diffusion Language Model [0.23332469289621785]
  We train an audio-native interface for DiffusionGemma that generates text by uniform, random-token discrete diffusion.<n>We find that the natural training objectives fail to ground the audio because their gradient reaches the projector only through attention that has already dismissed it.<n>The resulting model reaches 6.6 percent word error rate on LibriSpeech test-clean, transcribes in roughly eight parallel steps regardless of utterance length, and uses a single adapter trained on six languages.
  arXiv  Detail & Related papers  (2026-07-14T17:53:22Z)
• Unified Audio Intelligence Without Regressing on Text Intelligence [86.55418188552768]
  We introduce Nemotron-Labs-Audex-30B-A3B (Audex), a unified audio-text LLM.<n>Audex adopts a simple unified design with a single Transformer decoder.<n>Audex delivers state-of-the-art audio understanding, speech recognition and translation, text-to-speech, audio generation, and speech-to-speech generation.
  arXiv  Detail & Related papers  (2026-07-06T15:11:57Z)
• Do Audio LLMs Listen or Read? Analyzing and Mitigating Paralinguistic Failures with VoxParadox [4.088161686930475]
  We introduce VoxParadox, an adversarial benchmark with 2,000 verified examples, spanning 10 paralinguistic tasks.<n>We show consistently low accuracy on acoustic ground truth and a strong tendency to follow language-implied answers.<n>We propose Prompt-Conditioned Layer Mixer (PCLM), which adaptively combines information from multiple audio layers based on the input prompt.
  arXiv  Detail & Related papers  (2026-05-26T23:44:23Z)
• How Auditory Knowledge in LLM Backbones Shapes Audio Language Models: A Holistic Evaluation [97.0235251827591]
  Large language models (LLMs) have been widely used as knowledge backbones of Large Audio Language Models (LALMs)<n>We study how much auditory knowledge they encode through text-only pre-training and how this affects downstream performance.<n>Our findings reveal that auditory knowledge varies substantially across families, and text-only results are strongly correlated with audio performance.
  arXiv  Detail & Related papers  (2026-03-19T17:50:07Z)
• CS3-Bench: Evaluating and Enhancing Speech-to-Speech LLMs for Mandarin-English Code-Switching [31.584937435966253]
  We propose Code-Switching Speech-to-Speech Benchmark (CS3-Bench)<n>Experiments on 7 mainstream models demonstrate a relative performance drop of up to 66% in knowledge-intensive question answering.<n>Our approach improves the knowledge accuracy from 25.14% to 46.13%, with open-ended understanding rate from 64.5% to 86.5%, and significantly reduces pronunciation errors in the secondary language.
  arXiv  Detail & Related papers  (2025-10-09T07:34:23Z)
• Backdoor Attacks Against Speech Language Models [63.07317091368079]
  We present the first systematic study of audio backdoor attacks against speech language models.<n>We demonstrate its effectiveness across four speech encoders and three datasets, covering four tasks.<n>We propose a fine-tuning-based defense that mitigates the threat of poisoned pretrained encoders.
  arXiv  Detail & Related papers  (2025-10-01T17:45:04Z)
• What Makes a Good Speech Tokenizer for LLM-Centric Speech Generation? A Systematic Study [58.55905182336196]
  Speech-language models (SLMs) offer a promising path toward unifying speech and text understanding and generation.<n>We investigate the role of speech tokenizer designs in LLM-centric SLMs, augmented by speech heads and speaker modeling.<n>We introduce multi-token prediction (MTP) into SLMs, enabling each hidden state to decode multiple speech tokens.
  arXiv  Detail & Related papers  (2025-06-14T15:26:31Z)
• Reducing Object Hallucination in Large Audio-Language Models via Audio-Aware Decoding [54.82619273983179]
  Large Audio-Language Models (LALMs) can hallucinate what is presented in the audio.<n>We introduce Audio-Aware Decoding (AAD) to mitigate the hallucination of LALMs.<n>AAD uses contrastive decoding to compare the token prediction logits with and without the audio context.
  arXiv  Detail & Related papers  (2025-06-08T17:36:50Z)
• TransFace: Unit-Based Audio-Visual Speech Synthesizer for Talking Head Translation [54.155138561698514]
  Direct speech-to-speech translation achieves high-quality results through the introduction of discrete units obtained from self-supervised learning. Existing methods invariably rely on cascading, synthesizing via both audio and text, resulting in delays and cascading errors. We propose a model for talking head translation, textbfTransFace, which can directly translate audio-visual speech into audio-visual speech in other languages.
  arXiv  Detail & Related papers  (2023-12-23T08:45:57Z)
• Wav2vec-Switch: Contrastive Learning from Original-noisy Speech Pairs for Robust Speech Recognition [52.71604809100364]
  We propose wav2vec-Switch, a method to encode noise robustness into contextualized representations of speech. Specifically, we feed original-noisy speech pairs simultaneously into the wav2vec 2.0 network. In addition to the existing contrastive learning task, we switch the quantized representations of the original and noisy speech as additional prediction targets.
  arXiv  Detail & Related papers  (2021-10-11T00:08:48Z)
• The Interspeech Zero Resource Speech Challenge 2021: Spoken language modelling [19.525392906001624]
  We present the Zero Resource Speech Challenge 2021, which asks participants to learn a language model directly from audio, without any text or labels. The challenge is based on the Libri-light dataset, which provides up to 60k hours of audio from English audio books without any associated text.
  arXiv  Detail & Related papers  (2021-04-29T23:53:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.