Whisper Hallucination Detection and Mitigation via Hidden Representation Steering and Sparse AutoEncoders

• URL: http://arxiv.org/abs/2606.07473v1
• Date: Fri, 05 Jun 2026 17:26:23 GMT
• Title: Whisper Hallucination Detection and Mitigation via Hidden Representation Steering and Sparse AutoEncoders
• Abstract summary: We investigate whether hallucinations can be detected and mitigated through Whisper's internal representations.<n>We show that both spaces encode linearly separable hallucination-related information.<n>We propose two steering strategies: activation-space steering and SAE latent-space steering.
• Score: 7.378893412842889
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Whisper, a widely adopted ASR model, is known to suffer from hallucinations - coherent transcriptions generated for non-speech audio entirely disconnected from the input. We investigate whether hallucinations can be detected and mitigated through Whisper's internal representations. We extract audio encoder activations and evaluate two representation spaces: raw Whisper activations and Sparse AutoEncoder (SAE) latents. We show that both spaces encode linearly separable hallucination-related information, with discriminative power concentrated in a sparse feature subset and increasing toward deeper encoder layers. We propose two steering strategies: activation-space steering and SAE latent-space steering. SAE-based steering reduces hallucination rate from 72.63% to 14.11% for Whisper small and from 86.88% to 27.33% for Whisper large-v3 on the full non-speech test set, with small WER degradation on speech data, approaching the performance of fine-tuning-based methods.

Related papers

• Reducing Hallucinated Transcripts in Whisper via Hallucination Space Projection [3.1932005243797867]
  We propose a training-free, inference-time method to reduce hallucinations using low-rank projection of decoder activations.<n>We show that low-rank activation projection can substantially suppress Whisper hallucinations without retraining.
  arXiv  Detail & Related papers  (2026-09-03T23:33:12Z)
• Whisper-Aware LLM: Self-Supervised Uncertainty Learning for Robust Whispered Speech Recognition [7.1200667451435296]
  This paper introduces the Whisper-Aware LLM, a framework that teaches an Audio-LLM to perceive and react to whispered speech.<n>Our model develops an intrinsic self-awareness by learning to quantify the physical deficiencies of acoustic signals.<n>The model sets a new state-of-the-art on whispered speech with a 17% relative CER reduction on AISHELL6-Whisper.
  arXiv  Detail & Related papers  (2026-08-11T12:02:55Z)
• Steering Where to Listen: Instruction-Based Activation Steering Redirects Temporal Attention in Large Audio-Language Models [51.56484100374058]
  Large Audio-Language Models (LALMs) excel at audio understanding but expose little about where in an audio signal they attend.<n>We introduce instruction-based vector steering, which constructs a steering vector by contrasting activations from differently instructed prompts while keeping the audio fixed.<n>Our results characterize a mechanistic property of instruction-based steering in LALMs and provide a training-free probe for the latent temporal structure these models encode.
  arXiv  Detail & Related papers  (2026-06-09T19:44:46Z)
• Exploring Audio Hallucination in Egocentric Video Understanding [66.1760617001607]
  State-of-the-art large audio-visual language models (AV-LLMs) can generate multimodal descriptions.<n>We show in this work that they are prone to audio hallucinations, often inferring sounds from visual cues that are visible but not heard.
  arXiv  Detail & Related papers  (2026-04-26T20:06:58Z)
• Noise-Aware In-Context Learning for Hallucination Mitigation in ALLMs [5.553031534100783]
  Auditory large language models (ALLMs) have demonstrated strong general capabilities in audio understanding and reasoning tasks.<n>We propose a plug-and-play Noise-Aware In-Context Learning ( NAICL) method to address hallucination issues.
  arXiv  Detail & Related papers  (2026-04-10T06:35:46Z)
• AudioSAE: Towards Understanding of Audio-Processing Models with Sparse AutoEncoders [10.728025430620647]
  We train SAEs across all encoder layers of Whisper and HuBERT.<n>Over 50% of the features remain consistent across random seeds, and reconstruction quality is preserved.<n>We find SAE features correlated with human EEG activity during speech perception, indicating alignment with human neural processing.
  arXiv  Detail & Related papers  (2026-02-04T20:29:16Z)
• Listen Like a Teacher: Mitigating Whisper Hallucinations using Adaptive Layer Attention and Knowledge Distillation [9.486565210140279]
  The Whisper model is widely adopted for its strong performance across multilingual and zero-shot settings.<n>Previous works to reduce hallucinations in Whisper-style ASR systems have primarily focused on audio preprocessing or post-processing of transcriptions to filter out erroneous content.<n>We present a two-stage architecture that first enhances encoder robustness through Adaptive Layer Attention (ALA) and further suppresses hallucinations using a multi-objective knowledge distillation (KD) framework.
  arXiv  Detail & Related papers  (2025-11-18T07:52:47Z)
• Eliminating stability hallucinations in llm-based tts models via attention guidance [54.69225348397135]
  This paper focuses on resolving stability hallucinations in Text-to-Speech (TTS) models by improving and leveraging the attention mechanism.<n>We proposed a metric termed the Optimal Alignment Score (OAS), which employs the Viterbi algorithm to evaluate text-speech alignment quality.<n>The proposed methods can effectively reduce the stability of CosyVoice2 without introducing additional negative effects.
  arXiv  Detail & Related papers  (2025-09-24T07:47:52Z)
• SHALE: A Scalable Benchmark for Fine-grained Hallucination Evaluation in LVLMs [52.03164192840023]
  Large Vision-Language Models (LVLMs) still suffer from hallucinations, i.e., generating content inconsistent with input or established world knowledge.<n>We propose an automated data construction pipeline that produces scalable, controllable, and diverse evaluation data.<n>We construct SHALE, a benchmark designed to assess both faithfulness and factuality hallucinations.
  arXiv  Detail & Related papers  (2025-08-13T07:58:01Z)
• Calm-Whisper: Reduce Whisper Hallucination On Non-Speech By Calming Crazy Heads Down [9.098293248868503]
  We introduce a novel method to reduce Whisper's hallucination on non-speech segments without using pre- or post-possessing techniques.<n>We benchmark the contribution of each self-attentional head in the Whisper-large-v3 decoder to the hallucination problem by performing a head-wise mask.<n>Our best fine-tuned model, namely Calm-Whisper, achieves over 80% reduction in non-speech hallucination with only less than 0.1% WER.
  arXiv  Detail & Related papers  (2025-05-19T11:04:52Z)
• Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resampling [78.78822033285938]
  Vision-Language Models (VLMs) excel at visual understanding but often suffer from visual hallucinations.<n>In this work, we introduce REVERSE, a unified framework that integrates hallucination-aware training with on-the-fly self-verification.
  arXiv  Detail & Related papers  (2025-04-17T17:59:22Z)
• MLCA-AVSR: Multi-Layer Cross Attention Fusion based Audio-Visual Speech Recognition [62.89464258519723]
  We propose a multi-layer cross-attention fusion based AVSR approach that promotes representation of each modality by fusing them at different levels of audio/visual encoders. Our proposed approach surpasses the first-place system, establishing a new SOTA cpCER of 29.13% on this dataset.
  arXiv  Detail & Related papers  (2024-01-07T08:59:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.