Afrispeech Semantics: Evaluating Audio Semantic Reasoning in Spoken Language Models Across Domains and Accents

• URL: http://arxiv.org/abs/2606.11219v1
• Date: Mon, 11 May 2026 20:27:40 GMT
• Title: Afrispeech Semantics: Evaluating Audio Semantic Reasoning in Spoken Language Models Across Domains and Accents
• Abstract summary: Audio language models (ALMs) are increasingly used for speech-based understanding.<n>The effects of accent variation, domain shift, and semantic over-inference on audio reasoning are poorly understood.<n>We evaluate audio language models across five semantic and paralinguistic reasoning tasks.
• Score: 3.6734440150955368
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Audio language models (ALMs) are increasingly used for speech-based understanding, yet their ability to perform semantic reasoning beyond transcription, Text-to-Audio Retrieval, Captioning, and Question-Answering accuracy remains insufficiently benchmarked. In particular, the effects of accent variation, domain shift, and semantic over-inference on audio reasoning are poorly understood. We evaluate audio language models across five semantic and paralinguistic reasoning tasks: entailment, consistency, plausibility, accent drift, and accent restraint. Collectively, these tasks assess a model's ability to reason over spoken audio as the primary evidence source, including whether a textual hypothesis can be inferred, contradicted, or left undetermined by the audio, whether statements align or conflict with spoken content, whether claims are plausible given the discourse, and whether model predictions remain stable or appropriately constrained across accent variation. These findings highlight critical limitations in current audio reasoning evaluations and hope to provide guidance for more robust and equitable ALM design and assessment

Related papers

• When Text Misleads: Inconsistent-Aware Reasoning for Audio-Grounded Dialogue [18.262137202372454]
  We formalize failure mode as cross-modal disagreement, where transcripts suggest plausible but incorrect surface interpretations.<n>We develop a scalable framework that identifies text-biased surface interpretations and converts disagreement regions into conflict QA examples.<n>We further develop an agentic-style reasoning framework that converts speech into an Audio Twin, a text-readable representation of localized acoustic cues.
  arXiv  Detail & Related papers  (2026-08-27T14:26:46Z)
• X$^3$-OPD: Distilling Reasoning into Large Audio-Language Models via On-Policy Alignment [38.53954341285787]
  X$3$-OPD is a cross-modal on-policy distillation framework for large audio-language models.<n>It transfers reasoning capabilities from a powerful text teacher to an audio-language student.<n>X$3$-OPD substantially improves audio-grounded reasoning and chain-of-thought quality.
  arXiv  Detail & Related papers  (2026-07-23T17:35:20Z)
• Walking Through Uncertainty: An Empirical Study of Uncertainty Estimation for Audio-Aware Large Language Models [54.041320081289996]
  We present the first systematic empirical study of uncertainty estimation in audio-aware large language models (ALLMs)<n>We benchmark five representative methods, including predictive entropy, length-normalized entropy, semantic entropy, discrete semantic entropy, and P(True)<n>Our results reveal two key findings. First, semantic-level and verification-based methods consistently outperform token-level baselines on general audio reasoning benchmarks.
  arXiv  Detail & Related papers  (2026-04-28T12:56:22Z)
• DEAF: A Benchmark for Diagnostic Evaluation of Acoustic Faithfulness in Audio Language Models [15.637236155997]
  Recent Audio Multimodal Large Language Models (Audio MLLMs) demonstrate impressive performance on speech benchmarks.<n>It remains unclear whether these models genuinely process acoustic signals or rely on text-based semantic inference.<n>We introduce DEAF, a benchmark of over 2,700 conflict stimuli spanning three acoustic dimensions.
  arXiv  Detail & Related papers  (2026-03-17T15:52:26Z)
• On the Fallacy of Global Token Perplexity in Spoken Language Model Evaluation [88.77441715819366]
  Generative spoken language models pretrained on large-scale raw audio can continue a speech prompt with appropriate content.<n>We propose a variety of likelihood- and generative-based evaluation methods that serve in place of naive global token perplexity.
  arXiv  Detail & Related papers  (2026-01-09T22:01:56Z)
• Lost in Phonation: Voice Quality Variation as an Evaluation Dimension for Speech Foundation Models [22.710371114925763]
  Speech foundation models (SFMs) have enabled the direct processing of spoken language from raw audio, bypassing intermediate textual representations.<n>This capability allows SFMs to be exposed to, and potentially respond to, rich paralinguistic variations embedded in the input speech signal.<n>We introduce a new parallel dataset featuring synthesized modifications to voice quality, designed to evaluate SFM responses to creaky and breathy voice.
  arXiv  Detail & Related papers  (2025-10-29T14:44:44Z)
• Benchmarking Contextual and Paralinguistic Reasoning in Speech-LLMs: A Case Study with In-the-Wild Data [46.12417789276609]
  Speech-LLMs have shown impressive performance in tasks like transcription and translation, yet they remain limited in understanding the paralinguistic aspects of speech crucial for social and emotional intelligence.<n>We propose CP-Bench, a benchmark for evaluating speech-LLMs on contextual paralinguistic reasoning.
  arXiv  Detail & Related papers  (2025-09-20T09:26:40Z)
• SpeechR: A Benchmark for Speech Reasoning in Large Audio-Language Models [60.72029578488467]
  SpeechR is a unified benchmark for evaluating reasoning over speech in large audio-language models.<n>It evaluates models along three key dimensions: factual retrieval, procedural inference, and normative judgment.<n> Evaluations on eleven state-of-the-art LALMs reveal that high transcription accuracy does not translate into strong reasoning capabilities.
  arXiv  Detail & Related papers  (2025-08-04T03:28:04Z)
• Languages in Multilingual Speech Foundation Models Align Both Phonetically and Semantically [58.019484208091534]
  Cross-lingual alignment in pretrained language models (LMs) has enabled efficient transfer in text-based LMs.<n>It remains an open question whether findings and methods from text-based cross-lingual alignment apply to speech.
  arXiv  Detail & Related papers  (2025-05-26T07:21:20Z)
• The Model Hears You: Audio Language Model Deployments Should Consider the Principle of Least Privilege [48.18013944679755]
  Latest Audio Language Models (Audio LMs) process speech directly instead of relying on a separate transcription step.<n>This shift preserves detailed information, such as intonation or the presence of multiple speakers, that would otherwise be lost in transcription.<n>It also introduces new safety risks, including the potential misuse of speaker identity cues and other sensitive vocal attributes.
  arXiv  Detail & Related papers  (2025-03-21T04:03:59Z)
• Can Large Audio-Language Models Truly Hear? Tackling Hallucinations with Multi-Task Assessment and Stepwise Audio Reasoning [55.2480439325792]
  Large audio-language models (LALMs) have shown impressive capabilities in understanding and reasoning about audio and speech information.<n>These models still face challenges, including hallucinating non-existent sound events, misidentifying the order of sound events, and incorrectly attributing sound sources.
  arXiv  Detail & Related papers  (2024-10-21T15:55:27Z)
• VQMIVC: Vector Quantization and Mutual Information-Based Unsupervised Speech Representation Disentanglement for One-shot Voice Conversion [54.29557210925752]
  One-shot voice conversion can be effectively achieved by speech representation disentanglement. We employ vector quantization (VQ) for content encoding and introduce mutual information (MI) as the correlation metric during training. Experimental results reflect the superiority of the proposed method in learning effective disentangled speech representations.
  arXiv  Detail & Related papers  (2021-06-18T13:50:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.