Finding the Correct Visual Evidence Without Forgetting: Mitigating Hallucination in LVLMs via Inter-Layer Visual Attention Discrepancy

• URL: http://arxiv.org/abs/2605.20965v1
• Date: Wed, 20 May 2026 09:50:13 GMT
• Title: Finding the Correct Visual Evidence Without Forgetting: Mitigating Hallucination in LVLMs via Inter-Layer Visual Attention Discrepancy
• Abstract summary: We find that Large Vision-Language Models (LVLMs) tend to hallucinate when they pay insufficient attention to the correct visual evidence.<n>We propose a novel hallucination mitigation method that enhances visual evidence based on Inter-Layer Visual Attention Discrepancy (ILVAD)<n>Our method is training-free and plug-and-play.
• Score: 42.615995536459224
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Vision-Language Models (LVLMs) have shown remarkable performance on a wide range of vision-language tasks. Despite this progress, they are still prone to hallucination, generating responses that are inconsistent with visual content. In this work, we find that LVLMs tend to hallucinate when they pay insufficient attention to the correct visual evidence and gradually forget it during the generation process. We empirically find that although LVLMs overall attend insufficiently to visual evidence, they exhibit sensitivity to the correct visual evidence in specific layers, with notable inter-layer discrepancy. Motivated by this observation, we propose a novel hallucination mitigation method that enhances visual evidence based on Inter-Layer Visual Attention Discrepancy (ILVAD). Specifically, we obtain the attention weights from early generated tokens to visual tokens across layers and identify the tokens that are repeatedly activated as visual evidence, forming a saliency map. We then enhance attention to visual evidence during generation through the saliency map to reduce visual forgetting. In addition, we leverage the saliency map to obtain attention scores of generated text to visual evidence, in order to select and emphasize text tokens that are strongly grounded in visual evidence. Our method is training-free and plug-and-play. Multiple benchmark evaluations conducted on five recently released models show that our method can consistently mitigate hallucinations in different LVLMs over various architectures. Code is available at https://github.com/ytx-ML/ILVAD.

Related papers

• SeeMe: Mitigating Hallucinations in Large Vision-Language Models through Effective Visual Token Engineering [38.57574515804996]
  Large Vision-Language Models (LVLMs) have achieved remarkable progress in visual understanding tasks.<n>They remain susceptible to hallucinations, generating content that is inconsistent with the actual visual input.<n>We propose SeeMe, a training-free framework that introduces the concept of feature engineering into LVLMs.
  arXiv  Detail & Related papers  (2026-07-05T08:07:51Z)
• R-CoV: Region-Aware Chain-of-Verification for Alleviating Object Hallucinations in LVLMs [88.62912181680413]
  Region-aware Chain-of-Verification (R-CoV) is a visual chain-of-verification method to alleviate object hallucinations in large vision-language models.<n>R-CoV consists of six steps: initial response generation, entity extraction, coordinate generation, region description, verification execution, and final response generation.
  arXiv  Detail & Related papers  (2026-04-22T15:41:33Z)
• Visual Multi-Agent System: Mitigating Hallucination Snowballing via Visual Flow [99.54291580187817]
  Multi-Agent System (MAS) powered by Visual Language Models (VLMs) enables challenging tasks but suffers from a novel failure term, visual hallucination snowballing.<n>We provide detailed insights into the essence of hallucination snowballing regarding the reduction of visual attention allocation.<n>We propose ViF, a lightweight, plug-and-play mitigation paradigm that relays inter-agent messages with Visual Flow powered by the selected visual relay tokens and applies attention reallocation to amplify this pattern.
  arXiv  Detail & Related papers  (2025-09-26T02:43:24Z)
• See Different, Think Better: Visual Variations Mitigating Hallucinations in LVLMs [7.964168958699652]
  This paper presents ViHallu, a Vision-Centric Hallucination mitigation framework.<n>ViHallu introduces visual variation images with controllable visual alterations while maintaining the overall image structure.<n>Experiments show that ViHallu effectively enhances models' fine-grained visual understanding while significantly reducing hallucination tendencies.
  arXiv  Detail & Related papers  (2025-07-29T16:53:27Z)
• Attention Hijackers: Detect and Disentangle Attention Hijacking in LVLMs for Hallucination Mitigation [123.54980913741828]
  Large Vision-Language Models (LVLMs) remain vulnerable to hallucinations.<n>We propose a novel, training-free strategy namely Attention HIjackers Detection and Disentanglement (AID)<n>AID identifies Attention Hijackers by calculating instruction-driven visual salience.<n>Next, Attention Disentanglement mechanism is proposed to mask the visual attention of these identified Hijackers.<n>Re-Disentanglement recalculates the balance between instruction-driven and image-driven visual salience to avoid over-masking effects.
  arXiv  Detail & Related papers  (2025-03-11T09:35:55Z)
• Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens [7.806633929976787]
  Hallucinations in Large Vision-Language Models (LVLMs) significantly undermine their reliability.<n>This paper addresses how LVLMs process visual information and whether this process causes hallucination.<n>We propose a simple inference-time method that adjusts visual attention by integrating information across various heads.
  arXiv  Detail & Related papers  (2024-11-23T03:40:05Z)
• Mitigating Object Hallucination via Concentric Causal Attention [71.27325347912823]
  We show that object hallucination is closely tied with Rotary Position. RoPE, a widely adopted positional dependency modeling design. We propose Concentric Causal Attention (CCA), a simple yet effective positional alignment strategy.
  arXiv  Detail & Related papers  (2024-10-21T11:54:53Z)
• Visual Description Grounding Reduces Hallucinations and Boosts Reasoning in LVLMs [52.497823009176074]
  Large Vision-Language Models (LVLMs) often produce responses that misalign with factual information, a phenomenon known as hallucinations.<n>We introduce Visual Description Grounded Decoding (VDGD), a training-free method designed to enhance visual perception and improve reasoning capabilities in LVLMs.
  arXiv  Detail & Related papers  (2024-05-24T16:21:59Z)
• Pensieve: Retrospect-then-Compare Mitigates Visual Hallucination [14.25488878224697]
  We propose Pensieve, a training-free method that leverages the analogous visual hallucinations, which are induced by images sharing common semantic and appearance characteristics. Pensieve mitigates the effects of addressing errors from both the visual and textual branches by adaptively scaling the subtracted scores.
  arXiv  Detail & Related papers  (2024-03-21T13:49:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.