Adaptive vector steering: A training-free, layer-wise intervention for hallucination mitigation in large audio and multimodal models

• URL: http://arxiv.org/abs/2510.12851v1
• Date: Tue, 14 Oct 2025 08:52:18 GMT
• Title: Adaptive vector steering: A training-free, layer-wise intervention for hallucination mitigation in large audio and multimodal models
• Abstract summary: We propose Adaptive Vector Steering to better grounds generation in audio content.<n>Experiments show consistent performance gains across two models and two benchmarks.<n>To the best of our knowledge, this is the first work to apply vector steering to mitigate hallucination in audio.
• Score: 45.88028371034407
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Large Audio-Language Models and Multi-Modal Large Language Models have demonstrated strong capabilities in tasks such as Audio Question Answering (AQA), Audio Captioning, and Automatic Speech Recognition (ASR). However, there is growing evidence that these models can hallucinate about the content of the audio. To address this issue, we probe the models' internal states and propose Adaptive Vector Steering (AVS), a method that better grounds generation in audio content. We also identify a strong correlation between output correctness and internal representations. Experiments show consistent performance gains across two models and two benchmarks. On the Audio Hallucination QA dataset, our method boosts the F1-score of Gemma from 0.550 to 0.619 and Qwen from 0.626 to 0.632. Furthermore, our method increases the accuracy of Qwen on MMAU from 0.548 to 0.592, marking an 8% relative increase. To the best of our knowledge, this is the first work to apply vector steering to mitigate hallucination in audio.

Related papers

• TAD: Token-Adaptive Contrastive Decoding with Confidence-Guided Gating for Hallucination Mitigation in Large Audio-Language Models [9.497925590553857]
  Large audio-language models (LALMs) can hallucinate audio objects, answering "yes" to absent sound events.<n>We propose Token-Adaptive Decoding (TAD), a training-free strategy for hallucination mitigation.
  arXiv  Detail & Related papers  (2026-09-07T09:54:52Z)
• AMAuT: A Flexible and Efficient Multiview Audio Transformer Framework Trained from Scratch [0.3728263002609659]
  This paper introduces the Augmentation-driven Multiview Audio Transformer (AMAuT)<n>AMAuT eliminates the dependency on pre-trained weights while supporting arbitrary sample rates and audio lengths.<n> Experiments on five public benchmarks, AudioMNIST, SpeechCommands V1 & V2, VocalSound, and CochlScene, show that AMAuT achieves accuracies up to 99.8%.
  arXiv  Detail & Related papers  (2025-10-22T08:41:59Z)
• Audio-Maestro: Enhancing Large Audio-Language Models with Tool-Augmented Reasoning [45.88028371034407]
  We present Audio-Maestro -- a tool-augmented audio reasoning framework.<n>It enables audio-language models to autonomously call external tools and integrate their timestamped outputs into the reasoning process.<n>Experiments show that Audio-Maestro consistently improves general audio reasoning performance.
  arXiv  Detail & Related papers  (2025-10-13T14:25:34Z)
• Reducing Object Hallucination in Large Audio-Language Models via Audio-Aware Decoding [54.82619273983179]
  Large Audio-Language Models (LALMs) can hallucinate what is presented in the audio.<n>We introduce Audio-Aware Decoding (AAD) to mitigate the hallucination of LALMs.<n>AAD uses contrastive decoding to compare the token prediction logits with and without the audio context.
  arXiv  Detail & Related papers  (2025-06-08T17:36:50Z)
• Multi-Domain Audio Question Answering Toward Acoustic Content Reasoning in The DCASE 2025 Challenge [102.84031769492708]
  This task defines three QA subsets to test audio-language models on interactive question-answering over diverse acoustic scenes.<n>Preliminary results on the development set are compared, showing strong variation across models and subsets.<n>This challenge aims to advance the audio understanding and reasoning capabilities of audio-language models toward human-level acuity.
  arXiv  Detail & Related papers  (2025-05-12T09:04:16Z)
• Taming Data and Transformers for Audio Generation [31.815960560115176]
  AutoReCap-XL is the largest ambient audio-text dataset with over 47 million clips.<n>AutoCap is a high-quality automatic audio captioning model.<n>GenAu is a scalable transformer-based audio generation architecture.
  arXiv  Detail & Related papers  (2024-06-27T17:58:54Z)
• Audio Mamba: Bidirectional State Space Model for Audio Representation Learning [15.472819870523093]
  We introduce Audio Mamba, the first self-attention-free, purely SSM-based model for audio classification. We evaluate AuM on various audio datasets - comprising six different benchmarks - where it achieves comparable or better performance.
  arXiv  Detail & Related papers  (2024-06-05T15:00:59Z)
• Multilingual Audio-Visual Speech Recognition with Hybrid CTC/RNN-T Fast Conformer [59.57249127943914]
  We present a multilingual Audio-Visual Speech Recognition model incorporating several enhancements to improve performance and audio noise robustness. We increase the amount of audio-visual training data for six distinct languages, generating automatic transcriptions of unlabelled multilingual datasets. Our proposed model achieves new state-of-the-art performance on the LRS3 dataset, reaching WER of 0.8%.
  arXiv  Detail & Related papers  (2024-03-14T01:16:32Z)
• A Light Weight Model for Active Speaker Detection [7.253335671577093]
  We construct a lightweight active speaker detection architecture by reducing input candidates, splitting 2D and 3D convolutions for audio-visual feature extraction, and applying gated recurrent unit (GRU) with low computational complexity for cross-modal modeling. Experimental results on the AVA-ActiveSpeaker dataset show that our framework achieves competitive mAP performance (94.1% vs. 94.2%). Our framework also performs well on the Columbia dataset showing good robustness.
  arXiv  Detail & Related papers  (2023-03-08T08:40:56Z)
• AudioGen: Textually Guided Audio Generation [116.57006301417306]
  We tackle the problem of generating audio samples conditioned on descriptive text captions. In this work, we propose AaudioGen, an auto-regressive model that generates audio samples conditioned on text inputs.
  arXiv  Detail & Related papers  (2022-09-30T10:17:05Z)
• Large Scale Audiovisual Learning of Sounds with Weakly Labeled Data [9.072124914105325]
  We present an audiovisual fusion model that learns to recognize sounds from weakly labeled video recordings. Experiments on the large scale sound events dataset, AudioSet, demonstrate the efficacy of the proposed model.
  arXiv  Detail & Related papers  (2020-05-29T01:30:14Z)
• Audio ALBERT: A Lite BERT for Self-supervised Learning of Audio Representation [51.37980448183019]
  We propose Audio ALBERT, a lite version of the self-supervised speech representation model. We show that Audio ALBERT is capable of achieving competitive performance with those huge models in the downstream tasks. In probing experiments, we find that the latent representations encode richer information of both phoneme and speaker than that of the last layer.
  arXiv  Detail & Related papers  (2020-05-18T10:42:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.