Beyond Dominant Patches: Spatial Credit Redistribution For Grounded Vision-Language Models

• URL: http://arxiv.org/abs/2602.22469v2
• Date: Wed, 04 Mar 2026 18:21:09 GMT
• Title: Beyond Dominant Patches: Spatial Credit Redistribution For Grounded Vision-Language Models
• Abstract summary: Vision-Language Models (VLMs) often hallucinate objects that are not present in the input image.<n>We propose Spatial Credit Redistribution (SCR), a training-free-time inference method.<n>SCR achieves a better trade-off between hallucination reduction, generation quality, and latency.
• Score: 0.7919969809015935
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-Language Models (VLMs) often hallucinate objects that are not present in the input image. We identify a contributing cause of this behavior, which we term spatial credit collapse: in early transformer layers, hidden-state activation concentrates on a small number of visual patches, suppressing surrounding contextual evidence and increasing reliance on language priors. Across seven models we observe a strong correlation between visual attention entropy and hallucination rate (r = -0.65, p < 0.001), suggesting that reduced spatial credit diversity contributes to hallucination. To address this issue we propose Spatial Credit Redistribution (SCR), a training-free inference-time method. SCR uses a lightweight two-pass procedure. A diagnostic pass identifies the top-K high-attention source patches and their spatial neighbors. A redistribution pass then scales each source by 1/lambda (~0.91) and injects a (lambda - 1) weighted copy of its hidden state into neighboring patches, restoring suppressed visual context without modifying model weights. Because the diagnostic pass is performed once per image and reused across the output sequence, the added latency is negligible (<0.5 ms per token for 100-token responses). We evaluate SCR across seven model configurations from four VLM families (Chameleon, LLaVA-1.5, Qwen-VL/Qwen2-VL, and InternVL2) on five benchmarks: POPE, CHAIR, MME, HallusionBench, and AMBER. SCR reduces POPE-Adversarial hallucination by 4.6-6.0 percentage points and CHAIR-s by 41-51 percent while preserving caption quality (CIDEr drop <=0.8). Compared with prior inference-time methods including OPERA, VCD, OA-VCD, DoLa, VLI, SID, and CRoPS, SCR achieves a better trade-off between hallucination reduction, generation quality, and latency.

Related papers

• Beyond Language Priors: Diagnosing and Fixing Visual-Origin Hallucinations in Multimodal LLM [37.0916688511164]
  We present evidence for visual-origin hallucination, where hallucinations arise from incorrect visual feature extraction and misalignment between image and text embeddings.<n>We propose Adversarial Contrastive Fine-Tuning (ACFT) to correct this problem.<n>ACFT achieves state-of-the-art performance on POPE, MME, and four description-level hallucination benchmarks across LLaVA, MiniGPT-4, and Qwen2.5-VL.
  arXiv  Detail & Related papers  (2026-08-31T18:38:24Z)
• Hallucination Mitigation for Large Vision-Language Models via Implicit Feature Stabilization [11.087363456274737]
  Large Vision-Language Models (LVLMs) fluently describe objects, attributes, and scenes that are not in the image.<n>We connect part of this failure to a measurable property of their representations, feature instability, where mild semantics-preserving perturbations of the input cause large changes in the learned embeddings.<n>Existing stability-motivated remedies are explicit, in the sense that they intervene at inference time through latent steering or constrained decoding, and pay for it on every query.<n>We propose implicit stabilization instead: perturbation-invariance is built into the model weights during fine-tuning, and nothing extra runs at deployment
  arXiv  Detail & Related papers  (2026-08-30T17:39:38Z)
• See Only When Needed: Context-Aware Attention Intervention for Mitigating Hallucinations in LVLMs [81.71384150441163]
  We propose Context-aware Attention Intervention (CAI) for large vision-language models (LVLMs)<n>CAI enforces a see only when needed principle via two-axis selectivity: where to look and when to intervene.<n> experiments across multiple LVLM backbones and benchmarks show that CAI achieves state-of-the-art hallucination mitigation.
  arXiv  Detail & Related papers  (2026-06-29T06:35:47Z)
• To See or To Please: Uncovering Visual Sycophancy and Split Beliefs in VLMs [1.9349092719498848]
  Tri-Layer Diagnostic Framework disentangles hallucination sources via three metrics: Latent Anomaly Detection, Visual Necessity Score, and Competition Score.<n>Our taxonomy reveals that 69.6% of samples exhibit Visual Sycophancy--models detect visual anomalies but hallucinate to satisfy user expectations--while zero samples show Robust Refusal.
  arXiv  Detail & Related papers  (2026-03-19T00:15:05Z)
• CaTok: Taming Mean Flows for One-Dimensional Causal Image Tokenization [122.88484422855934]
  We present CaTok, a 1D causal image tokenizer with a MeanFlow decoder.<n>By selecting tokens over time intervals, CaTok learns causal 1D representations that support both fast one-step generation and high-fidelity multi-step sampling.<n> Experiments demonstrate that CaTok achieves state-of-the-art results on ImageNet reconstruction, reaching 0.75 FID, 22.53 PSNR and 0.674 SSIM.
  arXiv  Detail & Related papers  (2026-03-06T16:39:17Z)
• Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigation [51.743225614196774]
  Multimodal large language models (MLLMs) have achieved remarkable progress in vision-language reasoning.<n>They remain vulnerable to hallucination, where generated content deviates from visual evidence.<n>Recent vision enhancement methods attempt to address this issue by reinforcing visual tokens during decoding.<n>We propose Adaptive Visual Reinforcement (AIR), a training-free framework for MLLMs.
  arXiv  Detail & Related papers  (2026-02-27T14:18:51Z)
• Global Prior Meets Local Consistency: Dual-Memory Augmented Vision-Language-Action Model for Efficient Robotic Manipulation [95.89924101984566]
  We introduce OptimusVLA, a dual-memory VLA framework with Global Prior Memory (GPM) and Local Consistency Memory (LCM)<n>GPM replaces Gaussian noise with task-level priors retrieved from semantically similar trajectories.<n>LCM injects a learned consistency constraint that enforces temporal coherence and smoothness of trajectory.
  arXiv  Detail & Related papers  (2026-02-22T15:39:34Z)
• Hallucination Begins Where Saliency Drops [18.189047289404325]
  hallucinations frequently arise when preceding output tokens exhibit low saliency toward the prediction of the next token.<n>We introduce LVLMs-Saliency, a gradient-aware diagnostic framework that quantifies the visual grounding strength of each output token.<n>Our method significantly reduces hallucination rates while preserving fluency and task performance, offering a robust and interpretable solution.
  arXiv  Detail & Related papers  (2026-01-28T05:50:52Z)
• Gated Sparse Attention: Combining Computational Efficiency with Training Stability for Long-Context Language Models [0.0]
  Gated Sparse Attention (GSA) is an architecture that realizes the benefits of both sparse and gated attention.<n>GSA incorporates a gated lightning indexer with sigmoid activations that produce bounded, interpretable selection scores.
  arXiv  Detail & Related papers  (2026-01-12T20:33:39Z)
• Toward More Reliable Artificial Intelligence: Reducing Hallucinations in Vision-Language Models [0.0]
  Vision-language models (VLMs) frequently generate hallucinated content plausible but incorrect claims about image content.<n>We propose a training-free self-correction framework enabling VLMs to iteratively refine responses through uncertainty-guided visual re-attention.
  arXiv  Detail & Related papers  (2025-12-08T13:58:46Z)
• MapReduce LoRA: Advancing the Pareto Front in Multi-Preference Optimization for Generative Models [52.32146943039743]
  Reinforcement learning from human feedback has advanced alignment of generative models to human aesthetic and perceptual preferences.<n>We introduce two complementary methods: MapReduce LoRA and Reward-aware Token Embedding (RaTE)<n>Our framework sets a new state-of-the-art multi-preference alignment recipe across modalities.
  arXiv  Detail & Related papers  (2025-11-25T18:49:21Z)
• PruneHal: Reducing Hallucinations in Multi-modal Large Language Models through Adaptive KV Cache Pruning [87.35309934860938]
  hallucinations in large language models (MLLMs) are strongly associated with insufficient attention allocated to visual tokens.<n>We propose textbfPruneHal, a training-free, simple yet effective method that leverages adaptive KV cache pruning to enhance the model's focus on critical visual information.
  arXiv  Detail & Related papers  (2025-10-22T02:41:07Z)
• Token-Level Inference-Time Alignment for Vision-Language Models [58.41370989069588]
  Vision-Language Models (VLMs) have become essential backbones of modern multimodal intelligence.<n>We present TITA, a lightweight framework that freezes the base VLM and instead trains a reward model to approximate its distribution.<n>During inference, implicit preference signals are extracted as log-probability ratios between the reward model and the target VLM, yielding dense autoregressive feedback.
  arXiv  Detail & Related papers  (2025-10-20T09:58:03Z)
• Cross-Layer Attention Probing for Fine-Grained Hallucination Detection [6.83291363146574]
  We propose Cross-Layer Attention Probing (CLAP), a novel activation probing technique for hallucination detection.<n>Our empirical evaluations show that CLAP improves hallucination detection compared to baselines on both decoded responses and responses sampled at higher temperatures.<n>CLAP maintains high reliability even when applied out-of-distribution.
  arXiv  Detail & Related papers  (2025-09-04T14:37:34Z)
• Cure or Poison? Embedding Instructions Visually Alters Hallucination in Vision-Language Models [12.215295420714787]
  Vision-Language Models (VLMs) often suffer from hallucination, partly due to challenges in aligning multimodal information.<n>We propose Prompt-in-Image, a simple method that embeds textual instructions directly into images.<n>We evaluate this method on three popular open-source VLMs: Qwen2.5-VL, LLaVA-1.5, and InstructBLIP.
  arXiv  Detail & Related papers  (2025-08-03T09:11:18Z)
• SCISSOR: Mitigating Semantic Bias through Cluster-Aware Siamese Networks for Robust Classification [16.633948320306832]
  Shortcut learning undermines model generalization to out-of-distribution data.<n>We propose SCISSOR (Semantic Cluster Intervention for Suppressing ShORtcut), a network-based debiasing approach.<n>We evaluate SCISSOR on 6 models across 4 benchmarks: Chest-XRay and Not-MNIST in computer vision, and GYAFC and Yelp in NLP tasks.
  arXiv  Detail & Related papers  (2025-06-17T14:49:29Z)
• Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resampling [78.78822033285938]
  Vision-Language Models (VLMs) excel at visual understanding but often suffer from visual hallucinations.<n>In this work, we introduce REVERSE, a unified framework that integrates hallucination-aware training with on-the-fly self-verification.
  arXiv  Detail & Related papers  (2025-04-17T17:59:22Z)
• Benchmarking Reasoning Robustness in Large Language Models [76.79744000300363]
  We find significant performance degradation on novel or incomplete data.<n>These findings highlight the reliance on recall over rigorous logical inference.<n>This paper introduces a novel benchmark, termed as Math-RoB, that exploits hallucinations triggered by missing information to expose reasoning gaps.
  arXiv  Detail & Related papers  (2025-03-06T15:36:06Z)
• Mitigating Hallucination for Large Vision Language Model by Inter-Modality Correlation Calibration Decoding [66.06337890279839]
  Large vision-language models (LVLMs) have shown remarkable capabilities in visual-language understanding for downstream multi-modal tasks.<n>LVLMs still suffer from generating hallucinations in complex generation tasks, leading to inconsistencies between visual inputs and generated content.<n>We propose an Inter-Modality Correlation Decoding (IMCCD) method to mitigate hallucinations in LVLMs in a training-free manner.
  arXiv  Detail & Related papers  (2025-01-03T17:56:28Z)
• Mitigating Object Hallucination via Concentric Causal Attention [71.27325347912823]
  We show that object hallucination is closely tied with Rotary Position. RoPE, a widely adopted positional dependency modeling design. We propose Concentric Causal Attention (CCA), a simple yet effective positional alignment strategy.
  arXiv  Detail & Related papers  (2024-10-21T11:54:53Z)
• MM-Vet v2: A Challenging Benchmark to Evaluate Large Multimodal Models for Integrated Capabilities [146.4724093405187]
  We introduce MM-Vet v2, which includes a new "image-text sequence understanding" capability called "image-text sequence understanding"<n>Using MM-Vet v2 to benchmark large multimodal models, we found that Claude 3.5 Sonnet is the best model with a score of 71.8, slightly outperforming GPT-4o which scored 71.0.
  arXiv  Detail & Related papers  (2024-08-01T17:59:54Z)
• Coordinate Transformer: Achieving Single-stage Multi-person Mesh Recovery from Videos [91.44553585470688]
  Multi-person 3D mesh recovery from videos is a critical first step towards automatic perception of group behavior in virtual reality, physical therapy and beyond. We propose the Coordinate transFormer (CoordFormer) that directly models multi-person spatial-temporal relations and simultaneously performs multi-mesh recovery in an end-to-end manner. Experiments on the 3DPW dataset demonstrate that CoordFormer significantly improves the state-of-the-art, outperforming the previously best results by 4.2%, 8.8% and 4.7% according to the MPJPE, PAMPJPE, and PVE metrics, respectively.
  arXiv  Detail & Related papers  (2023-08-20T18:23:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.