SeeMe: Mitigating Hallucinations in Large Vision-Language Models through Effective Visual Token Engineering

• URL: http://arxiv.org/abs/2607.04163v1
• Date: Sun, 05 Jul 2026 08:07:51 GMT
• Title: SeeMe: Mitigating Hallucinations in Large Vision-Language Models through Effective Visual Token Engineering
• Abstract summary: Large Vision-Language Models (LVLMs) have achieved remarkable progress in visual understanding tasks.<n>They remain susceptible to hallucinations, generating content that is inconsistent with the actual visual input.<n>We propose SeeMe, a training-free framework that introduces the concept of feature engineering into LVLMs.
• Score: 38.57574515804996
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Vision-Language Models (LVLMs) have achieved remarkable progress in visual understanding tasks such as image captioning and visual question answering. However, they remain susceptible to hallucinations, generating content that is inconsistent with the actual visual input. Existing methods primarily intervene at the decoding stage, while overlooking a critical source of hallucinations: irrelevant or noisy visual tokens that mislead the decoding process. To address this issue, we propose SeeMe, a training-free framework that introduces the concept of feature engineering from traditional machine learning into LVLMs. SeeMe restructures visual tokens through a three-stage token engineering process to suppress hallucination sources while preserving informative visual evidence. Experiments on MME, POPE, and AMBER benchmarks across four LVLMs demonstrate that SeeMe consistently reduces hallucinations and improves output consistency, providing a novel perspective for mitigating hallucinations in LVLMs.

Related papers

• Finding the Correct Visual Evidence Without Forgetting: Mitigating Hallucination in LVLMs via Inter-Layer Visual Attention Discrepancy [42.615995536459224]
  We find that Large Vision-Language Models (LVLMs) tend to hallucinate when they pay insufficient attention to the correct visual evidence.<n>We propose a novel hallucination mitigation method that enhances visual evidence based on Inter-Layer Visual Attention Discrepancy (ILVAD)<n>Our method is training-free and plug-and-play.
  arXiv  Detail & Related papers  (2026-05-20T09:50:13Z)
• Residual Decoding: Mitigating Hallucinations in Large Vision-Language Models via History-Aware Residual Guidance [31.7541034166056]
  Large Vision-Language Models (LVLMs) can reason effectively from image-text inputs and perform well in various multimodal tasks.<n>They are affected by language priors and often produce hallucinations.<n>We propose Residual Decoding (ResDec) to address this problem.
  arXiv  Detail & Related papers  (2026-02-01T06:12:05Z)
• Enhancing Visual Reliance in Text Generation: A Bayesian Perspective on Mitigating Hallucination in Large Vision-Language Models [15.30139764717077]
  Large Vision-Language Models (LVLMs) usually generate texts which satisfy context coherence but don't match the visual input.<n>In this paper, we investigate the factors which may degenerate the visual reliance in text generation of LVLM from a Bayesian perspective.
  arXiv  Detail & Related papers  (2025-05-26T04:26:30Z)
• Cracking the Code of Hallucination in LVLMs with Vision-aware Head Divergence [69.86946427928511]
  We investigate the internal mechanisms driving hallucination in large vision-language models (LVLMs)<n>We introduce Vision-aware Head Divergence (VHD), a metric that quantifies the sensitivity of attention head outputs to visual context.<n>We propose Vision-aware Head Reinforcement (VHR), a training-free approach to mitigate hallucination by enhancing the role of vision-aware attention heads.
  arXiv  Detail & Related papers  (2024-12-18T15:29:30Z)
• CATCH: Complementary Adaptive Token-level Contrastive Decoding to Mitigate Hallucinations in LVLMs [74.36850397755572]
  CATCH addresses issues related to visual defects that cause diminished fine-grained feature perception and cumulative hallucinations in open-ended scenarios. It is applicable to various visual question-answering tasks without requiring any specific data or prior knowledge, and generalizes robustly to new tasks without additional training.
  arXiv  Detail & Related papers  (2024-11-19T18:27:31Z)
• From Pixels to Tokens: Revisiting Object Hallucinations in Large Vision-Language Models [15.401221354325672]
  Hallucinations in large vision models (LVLMs) are a significant challenge, i.e., generating objects that are not presented in the visual input.<n>Recent studies often attribute hallucinations to a lack of understanding of visual input, yet ignore a more fundamental issue: the model's inability to extract or decouple visual features.<n>In this paper, we revisit the hallucinations in LVLMs from an architectural perspective, investigating whether the primary cause lies in the visual encoder (feature extraction) or the modal alignment module (feature decoupling)
  arXiv  Detail & Related papers  (2024-10-09T11:46:32Z)
• Visual Description Grounding Reduces Hallucinations and Boosts Reasoning in LVLMs [52.497823009176074]
  Large Vision-Language Models (LVLMs) often produce responses that misalign with factual information, a phenomenon known as hallucinations.<n>We introduce Visual Description Grounded Decoding (VDGD), a training-free method designed to enhance visual perception and improve reasoning capabilities in LVLMs.
  arXiv  Detail & Related papers  (2024-05-24T16:21:59Z)
• Pensieve: Retrospect-then-Compare Mitigates Visual Hallucination [14.25488878224697]
  We propose Pensieve, a training-free method that leverages the analogous visual hallucinations, which are induced by images sharing common semantic and appearance characteristics. Pensieve mitigates the effects of addressing errors from both the visual and textual branches by adaptively scaling the subtracted scores.
  arXiv  Detail & Related papers  (2024-03-21T13:49:42Z)
• Hallucination Augmented Contrastive Learning for Multimodal Large Language Model [53.65682783591723]
  Multi-modal large language models (MLLMs) have been shown to efficiently integrate natural language with visual information to handle multi-modal tasks. However, MLLMs still face a fundamental limitation of hallucinations, where they tend to generate erroneous or fabricated information. In this paper, we address hallucinations in MLLMs from a novel perspective of representation learning.
  arXiv  Detail & Related papers  (2023-12-12T04:05:15Z)
• Evaluating Object Hallucination in Large Vision-Language Models [122.40337582958453]
  This work presents the first systematic study on object hallucination of large vision-language models (LVLMs) We find that LVLMs tend to generate objects that are inconsistent with the target images in the descriptions. We propose a polling-based query method called POPE to evaluate the object hallucination.
  arXiv  Detail & Related papers  (2023-05-17T16:34:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.