Auditing Generative Audio Calls for Known-Task Audio-LLM Evaluation

• URL: http://arxiv.org/abs/2608.27817v2
• Date: Mon, 31 Aug 2026 19:27:04 GMT
• Title: Auditing Generative Audio Calls for Known-Task Audio-LLM Evaluation
• Abstract summary: For known closed-set tasks, access to acoustic evidence and the need to call a generative audio model are important.<n>We evaluate this distinction as a controlled call-decision problem.<n>For known-task endpoint claims, the relevant quantity is the marginal value of the generative call after transcript and encoder evidence have already been used.
• Score: 8.345705067210861
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Speech and audio LLMs are often evaluated by asking whether a waveform prompt beats an automatic speech recognition (ASR) transcript. For known closed-set tasks, that comparison conflates two factors: access to acoustic evidence and the need to call a generative audio model. We evaluate this distinction as a controlled call-decision problem. For each example, a policy chooses among keeping a transcript label, using encoder evidence from Contrastive Language-Audio Pretraining (CLAP), Audio Spectrogram Transformer (AST), or WavLM, and calling Qwen2-Audio, Qwen2.5-Omni, or MOSS-Audio; the decisive ablation removes all generative actions while keeping the selector and development protocol fixed. On VocalSound, transcripts reach 0.296 accuracy, so waveform information is needed. Yet supervised CLAP and WavLM controls reach 0.850 and 0.854 with no generative audio calls. A selector with generative actions reaches 0.925 accuracy using 12.5% calls, compared with 0.921 for the matched no-call selector (paired difference 0.004; 95% CI [-0.025,0.033]). Agreement and stacking features improve weaker selectors but do not beat the strongest no-call control. For known-task endpoint claims, the relevant quantity is the marginal value of the generative call after transcript and encoder evidence have already been used.

Related papers

• Zero-Shot Temporal Localisation of Audio Deepfakes in Multi-Speaker Conversations [4.152774504513197]
  Training-free five-stage pipeline wraps a frozen binary detector and adds segment-level output with no retraining.<n>On 180 constructed multi-speaker conversations from ASVspoof 5, the system attains temporal intersection-over-union 0.90, temporal detection rate 0.95, and MS-DCF 0.26 with a strong backbone.<n>Its false-alarm rate on genuine speech is below 6%, falling under 2% on genuine real multi-speaker dialogue (AMI)<n>Under an identical pipeline, a trained localiser improves temporal IoU by only about 0.04, bounding the cost of forgoing supervision.
  arXiv  Detail & Related papers  (2026-09-09T11:27:26Z)
• TAD: Token-Adaptive Contrastive Decoding with Confidence-Guided Gating for Hallucination Mitigation in Large Audio-Language Models [9.497925590553857]
  Large audio-language models (LALMs) can hallucinate audio objects, answering "yes" to absent sound events.<n>We propose Token-Adaptive Decoding (TAD), a training-free strategy for hallucination mitigation.
  arXiv  Detail & Related papers  (2026-09-07T09:54:52Z)
• Auditing Generalization in AI-Generated Video Detection: A Six-Control Protocol and the VidAudit Toolkit [8.665845754660458]
  AI-generated video detection GenVidBench and AIGVDBench are the de facto leaderboards.<n>Most evaluation protocols leave uncontrolled confounds that can inflate reported generalizations.<n>A 20-paper survey finds none applying all six standard controls that would catch this.<n>We combine them into an audited protocol and apply it to six representative feature sources.
  arXiv  Detail & Related papers  (2026-06-30T00:29:53Z)
• TRADE: Transducer-Augmented Decoder for Speech LLM [52.69726809996728]
  Speech Large Language Models (Speech LLMs) lack a principled mechanism for streaming inference.<n>We propose TRADE TRansducer-Augmented DEcoder, which augments a multimodal LLM with a transducer branch that shares the audio encoder.<n>Three design choices make the system accurate, streamable, and long-form capable.
  arXiv  Detail & Related papers  (2026-06-07T07:15:34Z)
• Selective Attention System (SAS): Device-Addressed Speech Detection for Real-Time On-Device Voice AI [1.399948157377307]
  We show that, in multi-speaker environments with temporally ambiguous utterances, this task is more effectively modelled as a sequential routing problem over interaction history than as an utterance-local classification task.<n>We formalize this as Sequential Device-Addressed Routing (SDAR) and present the Selective Attention System (SAS), an on-device implementation that instantiates this formulation.
  arXiv  Detail & Related papers  (2026-04-09T16:11:15Z)
• VoiceSHIELD-Small: Real-Time Malicious Speech Detection and Transcription [2.0225826789157404]
  This paper introduces VoiceSHIELD-Small, a lightweight model that works in real time.<n>It can transcribe speech and detect whether it is safe or harmful, all in one step.<n>Built on OpenAI's Whisper-small encoder, VoiceSHIELD adds a mean-pooling layer and a simple classification head.
  arXiv  Detail & Related papers  (2026-03-08T16:23:42Z)
• Adaptive vector steering: A training-free, layer-wise intervention for hallucination mitigation in large audio and multimodal models [45.88028371034407]
  We propose Adaptive Vector Steering to better grounds generation in audio content.<n>Experiments show consistent performance gains across two models and two benchmarks.<n>To the best of our knowledge, this is the first work to apply vector steering to mitigate hallucination in audio.
  arXiv  Detail & Related papers  (2025-10-14T08:52:18Z)
• VoxRAG: A Step Toward Transcription-Free RAG Systems in Spoken Question Answering [4.740589102992697]
  We introduce VoxRAG, a modular speech-to-speech retrieval-augmented generation system.<n> VoxRAG bypasses transcription to retrieve semantically relevant audio segments directly from spoken queries.
  arXiv  Detail & Related papers  (2025-05-22T22:42:40Z)
• The Model Hears You: Audio Language Model Deployments Should Consider the Principle of Least Privilege [48.18013944679755]
  Latest Audio Language Models (Audio LMs) process speech directly instead of relying on a separate transcription step.<n>This shift preserves detailed information, such as intonation or the presence of multiple speakers, that would otherwise be lost in transcription.<n>It also introduces new safety risks, including the potential misuse of speaker identity cues and other sensitive vocal attributes.
  arXiv  Detail & Related papers  (2025-03-21T04:03:59Z)
• CLAIR-A: Leveraging Large Language Models to Judge Audio Captions [73.51087998971418]
  evaluating machine-generated audio captions is a complex task that requires considering diverse factors.<n>We propose CLAIR-A, a simple and flexible method that leverages the zero-shot capabilities of large language models.<n>In our evaluations, CLAIR-A better predicts human judgements of quality compared to traditional metrics.
  arXiv  Detail & Related papers  (2024-09-19T17:59:52Z)
• VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignment [101.2489492032816]
  VALL-E R is a robust and efficient zero-shot Text-to-Speech system. This research has the potential to be applied to meaningful projects, including the creation of speech for those affected by aphasia.
  arXiv  Detail & Related papers  (2024-06-12T04:09:44Z)
• Anomalous Sound Detection using Audio Representation with Machine ID based Contrastive Learning Pretraining [52.191658157204856]
  This paper uses contrastive learning to refine audio representations for each machine ID, rather than for each audio sample. The proposed two-stage method uses contrastive learning to pretrain the audio representation model. Experiments show that our method outperforms the state-of-the-art methods using contrastive learning or self-supervised classification.
  arXiv  Detail & Related papers  (2023-04-07T11:08:31Z)
• Using multiple reference audios and style embedding constraints for speech synthesis [68.62945852651383]
  The proposed model can improve the speech naturalness and content quality with multiple reference audios. The model can also outperform the baseline model in ABX preference tests of style similarity.
  arXiv  Detail & Related papers  (2021-10-09T04:24:29Z)
• A Second-Order Approach to Learning with Instance-Dependent Label Noise [58.555527517928596]
  The presence of label noise often misleads the training of deep neural networks. We show that the errors in human-annotated labels are more likely to be dependent on the difficulty levels of tasks.
  arXiv  Detail & Related papers  (2020-12-22T06:36:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.