Reducing Hallucinated Transcripts in Whisper via Hallucination Space Projection

• URL: http://arxiv.org/abs/2609.04561v1
• Date: Thu, 03 Sep 2026 23:33:12 GMT
• Title: Reducing Hallucinated Transcripts in Whisper via Hallucination Space Projection
• Abstract summary: We propose a training-free, inference-time method to reduce hallucinations using low-rank projection of decoder activations.<n>We show that low-rank activation projection can substantially suppress Whisper hallucinations without retraining.
• Score: 3.1932005243797867
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Whisper is a widely used foundation model for automatic speech recognition (ASR), but its generative decoder can produce fluent hallucinated transcripts for inputs containing little or no speech. We propose a training-free, inference-time method to reduce these hallucinations using low-rank projection of decoder activations. A compact hallucination-associated subspace is estimated from non-speech calibration data, and decoder hidden states are projected away from this subspace during inference. We evaluate two variants: always-on, which applies projection to all inputs, and gated, which applies it only when Whisper predicts that an input is likely non-speech. Across non-speech benchmarks, always-on projection reduces average hallucination rate (HR) from 31.31% to 2.44%, a 92.21% relative reduction, while gated projection reduces HR to 3.74%, an 88.05% relative reduction, with lower false rejection of genuine speech. On LibriSpeech, gated projection increases absolute word error rate (WER) by 0.33-4.39 percentage points and yields false-rejection rates (FRR) of 0.41--9.97% across model and split settings. These results show that low-rank activation projection can substantially suppress Whisper hallucinations without retraining, while providing a controllable trade-off between hallucination suppression and speech recognition performance.

Related papers

• Whisper Hallucination Detection and Mitigation via Hidden Representation Steering and Sparse AutoEncoders [7.378893412842889]
  We investigate whether hallucinations can be detected and mitigated through Whisper's internal representations.<n>We show that both spaces encode linearly separable hallucination-related information.<n>We propose two steering strategies: activation-space steering and SAE latent-space steering.
  arXiv  Detail & Related papers  (2026-06-05T17:26:23Z)
• Do LLM Decoders Listen Fairly? Benchmarking How Language Model Priors Shape Bias in Speech Recognition [9.131439623985028]
  Pretrained large language models replace task-specific decoders in speech recognition.<n>Do their text-derived priors make recognition fairer or more biased across demographic groups?<n>We evaluate nine models spanning three architectural generations on about 43,000 utterances across five demographic axes.
  arXiv  Detail & Related papers  (2026-04-23T04:40:58Z)
• The Semantic Illusion: Certified Limits of Embedding-Based Hallucination Detection in RAG Systems [0.0]
  We apply hallucination prediction to RAG detection, transforming scores into decision sets with finite-sample coverage guarantees.<n>We analyze this failure through the lens of distributional tails, showing that while NLI models achieve acceptable AUC (0.81), the "hardest" hallucinations are semantically indistinguishable from faithful responses.
  arXiv  Detail & Related papers  (2025-12-17T04:22:28Z)
• A Low-Rank Method for Vision Language Model Hallucination Mitigation in Autonomous Driving [18.863791958956625]
  Vision Language Models (VLMs) are increasingly used in autonomous driving to help understand traffic scenes.<n>VLMs produce hallucinations, which are false details not grounded in the visual input.<n>This paper proposes a novel self-contained low-rank approach to automatically rank multiple candidate captions based on their hallucination levels.
  arXiv  Detail & Related papers  (2025-11-09T18:50:30Z)
• StutterZero and StutterFormer: End-to-End Speech Conversion for Stuttering Transcription and Correction [0.0]
  This work introduces StutterZero and StutterFormer, the first end-to-end waveform-to-waveform models that directly convert stuttered speech into fluent speech.<n>Across all benchmarks, StutterZero had a 24% decrease in Word Error Rate (WER) and a 31% improvement in semantic similarity (BERTScore) compared to the leading Whisper-Medium model.
  arXiv  Detail & Related papers  (2025-10-21T17:54:36Z)
• Calm-Whisper: Reduce Whisper Hallucination On Non-Speech By Calming Crazy Heads Down [9.098293248868503]
  We introduce a novel method to reduce Whisper's hallucination on non-speech segments without using pre- or post-possessing techniques.<n>We benchmark the contribution of each self-attentional head in the Whisper-large-v3 decoder to the hallucination problem by performing a head-wise mask.<n>Our best fine-tuned model, namely Calm-Whisper, achieves over 80% reduction in non-speech hallucination with only less than 0.1% WER.
  arXiv  Detail & Related papers  (2025-05-19T11:04:52Z)
• Calibrating Verbal Uncertainty as a Linear Feature to Reduce Hallucinations [51.92795774118647]
  We find that verbal uncertainty'' is governed by a single linear feature in the representation space of LLMs.<n>We show that this has only moderate correlation with the actual semantic uncertainty'' of the model.
  arXiv  Detail & Related papers  (2025-03-18T17:51:04Z)
• ANAH-v2: Scaling Analytical Hallucination Annotation of Large Language Models [65.12177400764506]
  Large language models (LLMs) exhibit hallucinations in long-form question-answering tasks across various domains and wide applications.<n>Current hallucination detection and mitigation datasets are limited in domains and sizes.<n>This paper introduces an iterative self-training framework that simultaneously and progressively scales up the hallucination annotation dataset.
  arXiv  Detail & Related papers  (2024-07-05T17:56:38Z)
• Mitigating Object Hallucination in MLLMs via Data-augmented Phrase-level Alignment [52.43197107069751]
  Multimodal Large Language Models (MLLMs) often generate factually inaccurate information, referred to as hallucination.<n>We introduce Data-augmented Phrase-level Alignment (DPA), a novel loss which can be applied to instruction-tuned off-the-shelf MLLMs to mitigate hallucinations.
  arXiv  Detail & Related papers  (2024-05-28T23:36:00Z)
• Multi-Modal Hallucination Control by Visual Information Grounding [121.6983694815504]
  We show that Generative Vision-Language Models (VLMs) are prone to generate plausible-sounding textual answers that are not always grounded in the input image. We introduce Multi-Modal Mutual-Information Decoding (M3ID), a new sampling method for prompt amplification. M3ID amplifies the influence of the reference image over the language prior, hence favoring the generation of tokens with higher mutual information with the visual prompt.
  arXiv  Detail & Related papers  (2024-03-20T22:05:18Z)
• Fine-grained Hallucination Detection and Editing for Language Models [109.56911670376932]
  Large language models (LMs) are prone to generate factual errors, which are often called hallucinations. We introduce a comprehensive taxonomy of hallucinations and argue that hallucinations manifest in diverse forms. We propose a novel task of automatic fine-grained hallucination detection and construct a new evaluation benchmark, FavaBench.
  arXiv  Detail & Related papers  (2024-01-12T19:02:48Z)
• Alleviating Hallucinations of Large Language Models through Induced Hallucinations [67.35512483340837]
  Large language models (LLMs) have been observed to generate responses that include inaccurate or fabricated information. We propose a simple textitInduce-then-Contrast Decoding (ICD) strategy to alleviate hallucinations.
  arXiv  Detail & Related papers  (2023-12-25T12:32:49Z)
• A Stitch in Time Saves Nine: Detecting and Mitigating Hallucinations of LLMs by Validating Low-Confidence Generation [76.34411067299331]
  Large language models often tend to 'hallucinate' which critically hampers their reliability. We propose an approach that actively detects and mitigates hallucinations during the generation process. We show that the proposed active detection and mitigation approach successfully reduces the hallucinations of the GPT-3.5 model from 47.5% to 14.5% on average.
  arXiv  Detail & Related papers  (2023-07-08T14:25:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.