Wiener Representation Filtering for VLM Hallucination Suppression

• URL: http://arxiv.org/abs/2608.08167v1
• Date: Sat, 08 Aug 2026 14:51:17 GMT
• Title: Wiener Representation Filtering for VLM Hallucination Suppression
• Abstract summary: We propose a training-free, post-hoc representation editing technique that operates in the representation space of the language backbone.<n>By modeling hidden states as a superposition of truthful and hallucination filtering components, we derive a Wiener-type estimator.<n>Experiments on LLaVA1.53, MiniGPT-4, Gemma POPE, and mOwl2 demonstrate consistent reductions in object hallucination.
• Score: 45.33665443468439
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-language models (VLMs) excel at open-ended captioning and visual QA but often describe objects, attributes, or relations absent from the image, a phenomenon known as object hallucination. We propose a {training-free, post-hoc representation editing technique} that operates in the representation space of the language backbone. The method performs a lightweight, one-time offline calibration on a modest paired dataset to estimate the required covariance structures, using only forward passes and empirical second-order statistics with no gradient updates or fine-tuning, after which the correction is absorbed directly into the model's existing weights. By modeling hidden states as a superposition of truthful and hallucination-associated components, we derive a Wiener-type estimator whose optimal gains are given in closed form from the covariances of paired truthful and hallucinated representations. An eigendecomposition yields mode-wise attenuation that respects a stability criterion, i.e., the filter responds continuously to estimation noise. The correction is applied once to the feed-forward output projections of selected deeper layers, at inference time, the model runs unchanged and at the same speed. Experiments on LLaVA-1.5, MiniGPT-4, Gemma3, and mPLUG-Owl2 demonstrate consistent reductions in object hallucination on CHAIR, POPE, and MME while maintaining caption fluency and overall response quality. We further demonstrate the generality of our approach on the TempCompass video understanding benchmark and on discrete diffusion language models for grounded dialogue, showing that representation filtering reduces hallucinations even in temporal video reasoning and multi-step, sequence-wide denoising settings.

Related papers

• Dynamic Alignment Compensation for Hallucination Mitigation in Large Vision-Language Models [11.237547873256924]
  Large Vision-Language Models (LVLMs) remain prone to hallucinations, producing responses that are irrelevant or inconsistent with the multimodal input.<n>We propose emphDynamic Alignment Compensation (DAC), a training-free inference-time method that detects representation divergence and selectively applies residual compensation.<n>Experiments on nine hallucination-focused and general-purpose multimodal benchmarks across multiple LVLM backbones show that DAC consistently reduces hallucinations while maintaining strong overall performance.
  arXiv  Detail & Related papers  (2026-08-28T08:21:31Z)
• Beyond Global Editing: Per-Instance Disentangled Subspaces for Training-Free Hallucination Mitigation in LVLMs [16.83614016603383]
  We propose a training-free hallucination mitigation framework for large vision-language models.<n>Our method first constructs a set of Disentangled Hallucination Subspaces, each isolating a distinct hallucination mode.<n>Experiments across multiple vision-language benchmarks and LVLM families demonstrate consistent improvements.
  arXiv  Detail & Related papers  (2026-08-10T09:21:12Z)
• VCE: A zero-cost hallucination mitigation method of LVLMs via visual contrastive editing [70.82867621856968]
  Large vision-language models (LVLMs) frequently suffer from Object Hallucination (OH)<n>Recent studies suggest that the hallucination problem may stem from language priors.<n>We propose Visual Contrastive Editing (VCE), a novel post-hoc method that identifies and suppresses hallucinatory tendencies.
  arXiv  Detail & Related papers  (2026-04-21T12:40:07Z)
• VideoHEDGE: Entropy-Based Hallucination Detection for Video-VLMs via Semantic Clustering and Spatiotemporal Perturbations [4.509454543418357]
  Hallucinations in video-capable vision models (VideoVLMs) remain frequent and high-confidence.<n>We introduce VideoHEDGE, a modular framework for hallucination detection in question answering.
  arXiv  Detail & Related papers  (2026-01-13T13:42:05Z)
• Suppressing VLM Hallucinations with Spectral Representation Filtering [49.52703800684483]
  Vision-language models (VLMs) frequently produce hallucinations in the form of descriptions of objects, attributes, or relations that do not exist in the image.<n>We introduce Spectral Representation Filtering (SRF), a lightweight, training-free method to suppress such hallucinations by analyzing and correcting the covariance structure of the model's representations.
  arXiv  Detail & Related papers  (2025-11-15T13:49:27Z)
• Mitigating Hallucinations in Multimodal LLMs via Object-aware Preference Optimization [55.543583937522804]
  Multimodal Large Language Models (MLLMs) emerge as a unified interface to address a multitude of tasks.<n>Despite showcasing state-of-the-art results in many benchmarks, a long-standing issue is the tendency of MLLMs to hallucinate.<n>In this paper, we address the problem of hallucinations as an alignment problem, seeking to steer the MLLM so that it prefers generating content without hallucinations.
  arXiv  Detail & Related papers  (2025-08-27T18:02:04Z)
• Multimodal LLM-Guided Semantic Correction in Text-to-Image Diffusion [52.315729095824906]
  MLLM Semantic-Corrected Ping-Pong-Ahead Diffusion (PPAD) is a novel framework that introduces a Multimodal Large Language Model (MLLM) as a semantic observer during inference.<n>It performs real-time analysis on intermediate generations, identifies latent semantic inconsistencies, and translates feedback into controllable signals that actively guide the remaining denoising steps.<n>Extensive experiments demonstrate PPAD's significant improvements.
  arXiv  Detail & Related papers  (2025-05-26T14:42:35Z)
• OViP: Online Vision-Language Preference Learning for VLM Hallucination [44.14029765850719]
  Large vision-language models (LVLMs) remain vulnerable to hallucination, often generating content misaligned with visual inputs.<n>We propose an Online Vision-language Preference Learning framework that dynamically constructs contrastive training data based on the model's own hallucinations.
  arXiv  Detail & Related papers  (2025-05-21T19:26:09Z)
• Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resampling [78.78822033285938]
  Vision-Language Models (VLMs) excel at visual understanding but often suffer from visual hallucinations.<n>In this work, we introduce REVERSE, a unified framework that integrates hallucination-aware training with on-the-fly self-verification.
  arXiv  Detail & Related papers  (2025-04-17T17:59:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.