Hallucination Mitigation for Large Vision-Language Models via Implicit Feature Stabilization

• URL: http://arxiv.org/abs/2608.29924v1
• Date: Sun, 30 Aug 2026 17:39:38 GMT
• Title: Hallucination Mitigation for Large Vision-Language Models via Implicit Feature Stabilization
• Abstract summary: Large Vision-Language Models (LVLMs) fluently describe objects, attributes, and scenes that are not in the image.<n>We connect part of this failure to a measurable property of their representations, feature instability, where mild semantics-preserving perturbations of the input cause large changes in the learned embeddings.<n>Existing stability-motivated remedies are explicit, in the sense that they intervene at inference time through latent steering or constrained decoding, and pay for it on every query.<n>We propose implicit stabilization instead: perturbation-invariance is built into the model weights during fine-tuning, and nothing extra runs at deployment
• Score: 11.087363456274737
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Vision-Language Models (LVLMs) are prone to hallucinations: they fluently describe objects, attributes, and scenes that are not in the image. We connect part of this failure to a measurable property of their representations, feature instability, where mild semantics-preserving perturbations of the input cause large changes in the learned embeddings; hallucination rates rise together with this variability. Existing stability-motivated remedies are explicit, in the sense that they intervene at inference time through latent steering or constrained decoding, and pay for it on every query. We propose implicit stabilization instead: perturbation-invariance is built into the model weights during fine-tuning, and nothing extra runs at deployment. Our framework, INFUSE, first stabilizes visual and textual representations around perturbation-averaged and ground-truth anchors, then aligns the stabilized representations across modalities with bidirectional contrastive objectives. We prove that the anchor's root-mean-square deviation from the perturbation-mean representation shrinks at rate $1/\sqrt{K}$ in the number of views, and that under a Lipschitz decoder, this bounds how much any perturbation can change the model's hallucination behavior. On LLaVA-1.5, LLaVA-1.6, and Qwen3-VL-8B-Instruct, INFUSE reduces AMBER CHAIR by 46-63% relative to each base model, improves ObjHal, MMHal, HallusionBench, and POPE, and preserves VQA-v2 and TextVQA, all with no inference-time overhead.

Related papers

• Beyond Language Priors: Diagnosing and Fixing Visual-Origin Hallucinations in Multimodal LLM [37.0916688511164]
  We present evidence for visual-origin hallucination, where hallucinations arise from incorrect visual feature extraction and misalignment between image and text embeddings.<n>We propose Adversarial Contrastive Fine-Tuning (ACFT) to correct this problem.<n>ACFT achieves state-of-the-art performance on POPE, MME, and four description-level hallucination benchmarks across LLaVA, MiniGPT-4, and Qwen2.5-VL.
  arXiv  Detail & Related papers  (2026-08-31T18:38:24Z)
• Wiener Representation Filtering for VLM Hallucination Suppression [45.33665443468439]
  We propose a training-free, post-hoc representation editing technique that operates in the representation space of the language backbone.<n>By modeling hidden states as a superposition of truthful and hallucination filtering components, we derive a Wiener-type estimator.<n>Experiments on LLaVA1.53, MiniGPT-4, Gemma POPE, and mOwl2 demonstrate consistent reductions in object hallucination.
  arXiv  Detail & Related papers  (2026-08-08T14:51:17Z)
• MM-Snowball: Evaluating and Mitigating Hallucination Snowballing in Multimodal Multi-Turn Dialogue [98.58433916474472]
  We introduce MM-Snowball, the first benchmark for diagnosis of hallucination snowballing within dialogues.<n>We propose Conflict-Aware Visual Rectification (CAVR) to counteract this degradation.
  arXiv  Detail & Related papers  (2026-05-30T08:53:34Z)
• Mitigating Object Hallucinations in Vision-Language Models through Region-Aware Attention Recalibration [7.647944993041615]
  We introduce a training-free inference strategy to dynamically correct semantic drift without relying on abrupt truncations.<n>By computing an outlier-resistant statistical midpoint across various attention heads, we establish a stable anchor for reliable visual representations.<n>This recalibration process effectively rectifies visual-semantic misalignments while fully preserving generative fluency and language priors.
  arXiv  Detail & Related papers  (2026-05-24T09:14:49Z)
• Global Context or Local Detail? Adaptive Visual Grounding for Hallucination Mitigation [31.028607494171336]
  Vision-Language Models (VLMs) are frequently undermined by object hallucination.<n>We introduce Positive-and-Negative Decoding (PND), a training-free inference framework.<n>PND intervenes directly in the decoding process to enforce visual fidelity.
  arXiv  Detail & Related papers  (2026-04-27T12:23:00Z)
• VCE: A zero-cost hallucination mitigation method of LVLMs via visual contrastive editing [70.82867621856968]
  Large vision-language models (LVLMs) frequently suffer from Object Hallucination (OH)<n>Recent studies suggest that the hallucination problem may stem from language priors.<n>We propose Visual Contrastive Editing (VCE), a novel post-hoc method that identifies and suppresses hallucinatory tendencies.
  arXiv  Detail & Related papers  (2026-04-21T12:40:07Z)
• Learning Uncertainty from Sequential Internal Dispersion in Large Language Models [52.29267172760918]
  Uncertainty estimation is a promising approach to detect hallucinations in large language models.<n>Recent approaches commonly depend on model internal states to estimate uncertainty.<n>We present Sequential Internal Variance Representation (SIVR), a supervised hallucination detection framework.
  arXiv  Detail & Related papers  (2026-04-17T06:31:29Z)
• Same Answer, Different Representations: Hidden instability in VLMs [65.36933543377346]
  We introduce a representation-aware and frequency-aware evaluation framework that measures internal embedding drift, spectral sensitivity, and structural smoothness.<n>We apply this framework to modern Vision Language Models (VLMs) across the SEEDBench, MMMU, and POPE datasets.
  arXiv  Detail & Related papers  (2026-02-06T12:24:26Z)
• ReViP: Reducing False Completion in Vision-Language-Action Models with Vision-Proprioception Rebalance [50.05984919728878]
  We present ReViP, a novel VLA framework with Vision-Proprioception Rebalance to enhance visual grounding and robustness under perturbations.<n>Specifically, we use an external VLM as a task-stage observer to extract real-time task-centric visual cues from visual observations.<n>To evaluate false completion, we propose the first False-Completion Benchmark Suite built on LIBERO with controlled settings such as Object-Drop.
  arXiv  Detail & Related papers  (2026-01-23T11:31:07Z)
• Questioning the Stability of Visual Question Answering [11.848401203578456]
  Visual Language Models (VLMs) have achieved remarkable progress, yet their reliability under small, meaning-preserving input changes remains poorly understood.<n>We present the first large-scale, systematic study of VLM robustness to benign visual and textual perturbations.<n>We show that state-of-the-art systems (e.g., GPT-4o, Gemini 2.0 Flash) frequently fail under shifts as small as a few pixels or harmless rephrasings.
  arXiv  Detail & Related papers  (2025-11-14T12:05:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.