Auditing Protocol-Level Shortcuts in Large Audio Language Model Judges for Speech Evaluation

• URL: http://arxiv.org/abs/2607.13477v1
• Date: Wed, 15 Jul 2026 06:10:53 GMT
• Title: Auditing Protocol-Level Shortcuts in Large Audio Language Model Judges for Speech Evaluation
• Abstract summary: Large audio-language models (LALMs) are increasingly used as automatic judges for speech evaluation.<n>High agreement with human ratings does not guarantee that their verdicts are grounded in the audio.<n>We audit such protocol-level shortcuts'' in LALM judges across three common deployment protocols.
• Score: 35.33202492428442
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Large audio-language models (LALMs) are increasingly used as automatic judges for speech evaluation. However, high agreement with human ratings does not guarantee that their verdicts are grounded in the audio. A judge may instead rely on specialist labels or reference data supplied by the evaluation protocol itself, taking a shortcut in place of listening to the audio. In this paper, we audit such protocol-level ``shortcuts'' in LALM judges across three common deployment protocols: feature-blueprint judging, where the audio is replaced by a structured text description of acoustic features, reference-conditioned judging, and pairwise A/B comparison. Across six judges and four attributes, we find that several LALMs rely on protocol-level shortcuts. For example, in feature-blueprint judging, incorrect specialist labels reduce five judges' emotion accuracy to 0.10 or below, and in concatenated A/B comparisons, Qwen3-Omni-Thinking often picks the same slot regardless of order swaps. These results indicate that aggregate agreement can overstate the validity of LALM judges unless the model and the evaluation protocol are assessed jointly, and that each model-protocol pair should be evaluated with a matched shortcut probe.

Related papers

• Do Audio Language Models Use Paralinguistic Evidence? Counterfactual Audits for Response Evaluation [21.291909657115607]
  We introduce counterfactual audits for paralinguistic response evaluation.<n>Each audit item holds the transcript fixed while varying affect, prosody, or the timing of an affective shift.<n>This yields useful diagnostic states that identify different sources of judge failures.
  arXiv  Detail & Related papers  (2026-08-07T02:24:12Z)
• Improving Text-to-Audio Instruction Following via Fine-Grained Feedback from Audio-Aware Large Language Models [51.75139461584678]
  Recent text-to-audio models generate high-quality audio, but often fail to follow instructions involving multiple sound events and temporal order.<n>We propose an instruction-level framework that uses audio-aware large language models as fine-grained judges to verify target event presence and temporal relations.<n>Experiments show that our method improves event completeness, temporal ordering, and joint instruction-following accuracy across existing benchmarks and S3Bench, while maintaining audio quality.
  arXiv  Detail & Related papers  (2026-07-15T03:13:06Z)
• ParaPairAudioBench: Paralinguistic Pairwise Audio Benchmark for LALM-as-a-Judge [30.520810407664072]
  ParaPairAudioBench is a pairwise benchmark of 5,175 audio pairs across five paralinguistic dimensions: Style, Rate, Emphasis, Age, and Gender.<n>Experiments show that current LALM judges still lag behind human judgments by 32%p on average and exhibit severe calibration failures.
  arXiv  Detail & Related papers  (2026-06-23T14:43:43Z)
• Does the Judge Prefer English? Evaluating Language-Switching Invariance in LLM-as-a-Judge [0.0]
  We propose a lightweight meta-evaluation protocol that transforms LLMBar response-pair items into English, Chinese, and Chinese-English language-switched variants.<n>A reliable judge should preserve its preference under label-accessible language transformations.<n>Across models, Chinese and language-switched presentations induce 10.7--14.4% preference flips relative to English, and all judges achieve their highest accuracy in English.
  arXiv  Detail & Related papers  (2026-06-12T08:59:26Z)
• SAM Audio Judge: A Unified Multimodal Framework for Perceptual Evaluation of Audio Separation [52.468945848774844]
  This paper addresses the need for automated systems capable of evaluating audio separation without human intervention.<n>The proposed evaluation metric, SAM Audio Judge (SAJ), is a multimodal fine-grained reference-free objective metric.<n>SAJ supports three audio domains (speech, music and general sound events) and three prompt inputs (text, visual and span), covering four different dimensions of evaluation.
  arXiv  Detail & Related papers  (2026-01-27T15:29:02Z)
• Hearing Between the Lines: Unlocking the Reasoning Power of LLMs for Speech Evaluation [19.92868268408954]
  Large Language Model (LLM) judges exhibit strong reasoning capabilities but are limited to textual content.<n>We propose TRACE, a novel framework that enables LLM judges to reason over audio cues to achieve cost-efficient and human-aligned S2S evaluation.<n>We will release the HCoT annotations and the TRACE framework to enable scalable and human-aligned S2S evaluation.
  arXiv  Detail & Related papers  (2026-01-20T08:57:02Z)
• AHELM: A Holistic Evaluation of Audio-Language Models [78.20477815156484]
  multimodal audio-language models (ALMs) take interleaved audio and text as input and output text.<n>AHELM is a benchmark that aggregates various datasets -- including 2 new synthetic audio-text datasets called PARADE and CoRe-Bench.<n>We also standardize the prompts, inference parameters, and evaluation metrics to ensure equitable comparisons across models.
  arXiv  Detail & Related papers  (2025-08-29T07:40:39Z)
• Validating LLM-as-a-Judge Systems under Rating Indeterminacy [65.137380612741]
  We introduce a framework for validating LLM-as-a-judge systems under rating indeterminacy.<n>We demonstrate that differences in how humans and LLMs resolve rating indeterminacy when responding to forced-choice rating instructions can heavily bias validation.
  arXiv  Detail & Related papers  (2025-03-07T22:09:47Z)
• CLAIR-A: Leveraging Large Language Models to Judge Audio Captions [73.51087998971418]
  evaluating machine-generated audio captions is a complex task that requires considering diverse factors.<n>We propose CLAIR-A, a simple and flexible method that leverages the zero-shot capabilities of large language models.<n>In our evaluations, CLAIR-A better predicts human judgements of quality compared to traditional metrics.
  arXiv  Detail & Related papers  (2024-09-19T17:59:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.