When Looking Is Not Enough: Visual Attention Structure Reveals Hallucination in MLLMs

• URL: http://arxiv.org/abs/2605.11559v1
• Date: Tue, 12 May 2026 05:42:11 GMT
• Title: When Looking Is Not Enough: Visual Attention Structure Reveals Hallucination in MLLMs
• Abstract summary: We propose LaSCD (Laplacian-Spectral Contrastive Decoding), a training-free decoding strategy that selects informative layers via Laplacian energy and remaps next-token logits in closed form.<n> Experiments on hallucination and general multimodal benchmarks show that LaSCD consistently reduces hallucination while preserving general capabilities.
• Score: 43.42932571168595
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal large language models (MLLMs) have become a key interface for visual reasoning and grounded question answering, yet they remain vulnerable to visual hallucinations, where generated responses contradict image content or mention nonexistent objects. A central challenge is that hallucination is not always caused by a simple lack of visual attention: the model may still assign substantial attention mass to image tokens while internally drifting toward an incorrect answer. In this paper, we show that the high-frequency structure of visual attention, measured by layer-wise Laplacian energy, reveals both the layer where hallucinated preferences emerge and the layer where the ground-truth answer transiently recovers. Building on this finding, we propose LaSCD (Laplacian-Spectral Contrastive Decoding), a training-free decoding strategy that selects informative layers via Laplacian energy and remaps next-token logits in closed form. Experiments on hallucination and general multimodal benchmarks show that LaSCD consistently reduces hallucination while preserving general capabilities, highlighting its potential as a faithful decoding paradigm. The code is available at https://github.com/macovaseas/LaSCD.

Related papers

• Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucination [23.128973540926552]
  We find that both real and hallucinated objects receive equally strong visual attention in the model's mid-to-late layers.<n>We propose a training-free Detect-Mitigate framework comprising a Logit-Lens Consistency Check to detect hallucination.
  arXiv  Detail & Related papers  (2026-08-07T14:56:12Z)
• VEGAS: Mitigating Hallucinations in Large Vision-Language Models via Vision-Encoder Attention Guided Adaptive Steering [5.541436522468184]
  Large vision-language models (LVLMs) produce outputs that are linguistically fluent but factually inconsistent with the visual evidence.<n>We show that LVLMs tend to hallucinate when their final visual-attention maps fail to concentrate on key image objects.<n>We introduce VEGAS, a method that integrates the vision encoder's attention maps into the language model's mid-layers and adaptively steers tokens which fail to concentrate on key image objects.
  arXiv  Detail & Related papers  (2025-12-12T23:33:50Z)
• Mitigating Hallucination in Multimodal LLMs with Layer Contrastive Decoding [18.980167452015966]
  We propose a simple approach called Layer Contrastive Decoding (LayerCD)<n>LayerCD aims to filter out hallucinations by contrasting the output distributions generated from visual features of different levels.<n>We conduct extensive experiments on two benchmarks and show that LayerCD significantly outperforms current state-of-the-art.
  arXiv  Detail & Related papers  (2025-09-29T17:59:16Z)
• Mitigating Hallucinations in Vision-Language Models through Image-Guided Head Suppression [16.465703269782654]
  Large vision language models (LVLMs) often suffer from hallucinations, generating texts misaligned with the visual context.<n>Existing methods aimed at reducing hallucinations through inference time intervention incur a significant increase in latency.<n>We present SPIN, a task-agnostic attention-guided head suppression strategy that can be seamlessly integrated during inference.
  arXiv  Detail & Related papers  (2025-05-22T09:00:57Z)
• TARAC: Mitigating Hallucination in LVLMs via Temporal Attention Real-time Accumulative Connection [6.006482486396196]
  We propose Temporal Attention Real-time Accumulative Connection (TARAC) to mitigate hallucinations caused by the decay of attention on image tokens.<n>We validate TARAC across multiple models and datasets, demonstrating that our approach substantially mitigates hallucinations.
  arXiv  Detail & Related papers  (2025-04-05T07:57:11Z)
• Trust Me, I'm Wrong: LLMs Hallucinate with Certainty Despite Knowing the Answer [51.7407540261676]
  We investigate a distinct type of hallucination, where a model can consistently answer a question correctly, but a seemingly trivial perturbation causes it to produce a hallucinated response with high certainty.<n>This phenomenon is particularly concerning in high-stakes domains such as medicine or law, where model certainty is often used as a proxy for reliability.<n>We show that CHOKE examples are consistent across prompts, occur in different models and datasets, and are fundamentally distinct from other hallucinations.
  arXiv  Detail & Related papers  (2025-02-18T15:46:31Z)
• Towards a Systematic Evaluation of Hallucinations in Large-Vision Language Models [57.58426038241812]
  Large Vision-Language Models (LVLMs) have demonstrated remarkable performance in complex multimodal tasks.<n>These models still suffer from hallucinations when required to implicitly recognize or infer diverse visual entities from images.<n>We propose a novel visual question answering (VQA) benchmark that employs contextual reasoning prompts as hallucination attacks.
  arXiv  Detail & Related papers  (2024-12-29T23:56:01Z)
• Mitigating Object Hallucination via Concentric Causal Attention [71.27325347912823]
  We show that object hallucination is closely tied with Rotary Position. RoPE, a widely adopted positional dependency modeling design. We propose Concentric Causal Attention (CCA), a simple yet effective positional alignment strategy.
  arXiv  Detail & Related papers  (2024-10-21T11:54:53Z)
• Does Object Grounding Really Reduce Hallucination of Large Vision-Language Models? [53.89380284760555]
  Large vision-language models (LVLMs) produce captions that mention concepts that cannot be found in the image. These hallucinations erode the trustworthiness of LVLMs and are arguably among the main obstacles to their ubiquitous adoption. Recent work suggests that addition of grounding objectives -- those that explicitly align image regions or objects to text spans -- reduces the amount of LVLM hallucination.
  arXiv  Detail & Related papers  (2024-06-20T16:56:11Z)
• AutoHallusion: Automatic Generation of Hallucination Benchmarks for Vision-Language Models [91.78328878860003]
  Large vision-language models (LVLMs) are prone to hallucinations. benchmarks often rely on hand-crafted corner cases whose failure patterns may not generalize well. We develop AutoHallusion, the first automated benchmark generation approach.
  arXiv  Detail & Related papers  (2024-06-16T11:44:43Z)
• Visual Description Grounding Reduces Hallucinations and Boosts Reasoning in LVLMs [52.497823009176074]
  Large Vision-Language Models (LVLMs) often produce responses that misalign with factual information, a phenomenon known as hallucinations.<n>We introduce Visual Description Grounded Decoding (VDGD), a training-free method designed to enhance visual perception and improve reasoning capabilities in LVLMs.
  arXiv  Detail & Related papers  (2024-05-24T16:21:59Z)
• Logical Closed Loop: Uncovering Object Hallucinations in Large Vision-Language Models [52.957842999317506]
  Object hallucination refers to the phenomenon that the LVLMs claim non-existent objects in the image. We propose a Logical Closed Loop-based framework for Object Hallucination Detection and Mitigation, namely LogicCheckGPT. As a plug-and-play method, it can be seamlessly applied to all existing LVLMs.
  arXiv  Detail & Related papers  (2024-02-18T15:28:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.