Beyond the Global Scores: Fine-Grained Token Grounding as a Robust Detector of LVLM Hallucinations

• URL: http://arxiv.org/abs/2604.04863v1
• Date: Mon, 06 Apr 2026 17:07:11 GMT
• Title: Beyond the Global Scores: Fine-Grained Token Grounding as a Robust Detector of LVLM Hallucinations
• Abstract summary: Large vision-language models (LVLMs) achieve strong performance on visual reasoning tasks but remain highly susceptible to hallucination.<n>Existing detection methods rely on coarse, whole-image measures of how an object token relates to the input image.<n>We introduce a patch-level hallucination detection framework that examines fine-grained token-level interactions across model layers.<n>Our approach achieves up to 90% accuracy in token-level hallucination detection, demonstrating the superiority of fine-grained structural analysis for detecting hallucinations.
• Score: 32.1925100166762
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Large vision-language models (LVLMs) achieve strong performance on visual reasoning tasks but remain highly susceptible to hallucination. Existing detection methods predominantly rely on coarse, whole-image measures of how an object token relates to the input image. This global strategy is limited: hallucinated tokens may exhibit weak but widely scattered correlations across many local regions, which aggregate into deceptively high overall relevance, thus evading the current global hallucination detectors. We begin with a simple yet critical observation: a faithful object token must be strongly grounded in a specific image region. Building on this insight, we introduce a patch-level hallucination detection framework that examines fine-grained token-level interactions across model layers. Our analysis uncovers two characteristic signatures of hallucinated tokens: (i) they yield diffuse, non-localized attention patterns, in contrast to the compact, well-focused attention seen in faithful tokens; and (ii) they fail to exhibit meaningful semantic alignment with any visual region. Guided by these findings, we develop a lightweight and interpretable detection method that leverages patch-level statistical features, combined with hidden-layer representations. Our approach achieves up to 90% accuracy in token-level hallucination detection, demonstrating the superiority of fine-grained structural analysis for detecting hallucinations.

Related papers

• See Only When Needed: Context-Aware Attention Intervention for Mitigating Hallucinations in LVLMs [81.71384150441163]
  We propose Context-aware Attention Intervention (CAI) for large vision-language models (LVLMs)<n>CAI enforces a see only when needed principle via two-axis selectivity: where to look and when to intervene.<n> experiments across multiple LVLM backbones and benchmarks show that CAI achieves state-of-the-art hallucination mitigation.
  arXiv  Detail & Related papers  (2026-06-29T06:35:47Z)
• Scalable Token-Level Hallucination Detection in Large Language Models [63.3426544914783]
  Internal hallucinations are difficult to detect in reasoning-intensive tasks.<n>We propose TokenHD, a holistic pipeline for training token-level hallucination detectors.
  arXiv  Detail & Related papers  (2026-05-12T16:47:40Z)
• Cognitive Pivot Points and Visual Anchoring: Unveiling and Rectifying Hallucinations in Multimodal Reasoning Models [65.57985131861399]
  hallucinations are strongly correlated with cognitive bifurcation points that often exhibit high entropy states.<n>We propose V-STAR, a lightweight, holistic training paradigm designed to internalize visually aware reasoning capabilities.
  arXiv  Detail & Related papers  (2026-04-11T13:59:05Z)
• Hallucination Begins Where Saliency Drops [18.189047289404325]
  hallucinations frequently arise when preceding output tokens exhibit low saliency toward the prediction of the next token.<n>We introduce LVLMs-Saliency, a gradient-aware diagnostic framework that quantifies the visual grounding strength of each output token.<n>Our method significantly reduces hallucination rates while preserving fluency and task performance, offering a robust and interpretable solution.
  arXiv  Detail & Related papers  (2026-01-28T05:50:52Z)
• Revealing Perception and Generation Dynamics in LVLMs: Mitigating Hallucinations via Validated Dominance Correction [59.801614364841775]
  Large Vision-Language Models (LVLMs) have shown remarkable capabilities, yet hallucinations remain a persistent challenge.<n>This work presents a systematic analysis of the internal evolution of visual perception and token generation in LVLMs.<n>We devise the VDC (d Dominance Correction) strategy, which detects unsupported tokens and replaces them with validated ones to improve output reliability.
  arXiv  Detail & Related papers  (2025-12-21T17:05:42Z)
• The Map of Misbelief: Tracing Intrinsic and Extrinsic Hallucinations Through Attention Patterns [1.0896567381206717]
  Large Language Models (LLMs) are increasingly deployed in safety-critical domains, yet remain susceptible to hallucinations.<n>We introduce a principled evaluation framework that differentiates between extrinsic and intrinsic hallucination categories.<n>We propose novel attention aggregation strategies that improve both interpretability and hallucination detection performance.
  arXiv  Detail & Related papers  (2025-11-13T22:42:18Z)
• On Epistemic Uncertainty of Visual Tokens for Object Hallucinations in Large Vision-Language Models [27.228426342808486]
  We argue that uncertain visual tokens within the vision encoder (VE) is a key factor that contributes to object hallucination.<n>We propose a simple yet effective strategy to mitigate object hallucination by modifying the VE only.
  arXiv  Detail & Related papers  (2025-10-10T05:12:52Z)
• GLSim: Detecting Object Hallucinations in LVLMs via Global-Local Similarity [6.744353807473373]
  We introduce GLSim, a training-free object hallucination detection framework.<n>We demonstrate that GLSim achieves superior detection performance, outperforming competitive baselines by a significant margin.
  arXiv  Detail & Related papers  (2025-08-27T15:30:06Z)
• SHALE: A Scalable Benchmark for Fine-grained Hallucination Evaluation in LVLMs [52.03164192840023]
  Large Vision-Language Models (LVLMs) still suffer from hallucinations, i.e., generating content inconsistent with input or established world knowledge.<n>We propose an automated data construction pipeline that produces scalable, controllable, and diverse evaluation data.<n>We construct SHALE, a benchmark designed to assess both faithfulness and factuality hallucinations.
  arXiv  Detail & Related papers  (2025-08-13T07:58:01Z)
• When Semantics Mislead Vision: Mitigating Large Multimodal Models Hallucinations in Scene Text Spotting and Understanding [75.57997630182136]
  We investigate the underlying causes of semantic hallucination and identify a key finding: Transformer layers in Large Multimodal Models with stronger attention focus on scene text regions are less prone to producing semantic hallucinations.<n>We propose a training-free semantic hallucination mitigation framework comprising two key components: ZoomText and Grounded Layer Correction.<n>Our method not only effectively mitigates semantic hallucination but also achieves strong performance on public benchmarks for scene text spotting and understanding.
  arXiv  Detail & Related papers  (2025-06-05T19:53:19Z)
• CHAIR -- Classifier of Hallucination as Improver [1.397828249435483]
  We introduce CHAIR (Classifier of Hallucination As ImproveR), a supervised framework for detecting hallucinations by analyzing internal logits from each layer of every token.<n>Our method extracts a compact set of features such as maximum, minimum, mean, standard deviation, and slope-from the token logits across all layers, enabling effective hallucination detection without overfitting.
  arXiv  Detail & Related papers  (2025-01-05T12:15:02Z)
• Enhancing Uncertainty-Based Hallucination Detection with Stronger Focus [99.33091772494751]
  Large Language Models (LLMs) have gained significant popularity for their impressive performance across diverse fields. LLMs are prone to hallucinate untruthful or nonsensical outputs that fail to meet user expectations. We propose a novel reference-free, uncertainty-based method for detecting hallucinations in LLMs.
  arXiv  Detail & Related papers  (2023-11-22T08:39:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.