Global Context or Local Detail? Adaptive Visual Grounding for Hallucination Mitigation

• URL: http://arxiv.org/abs/2604.24396v1
• Date: Mon, 27 Apr 2026 12:23:00 GMT
• Title: Global Context or Local Detail? Adaptive Visual Grounding for Hallucination Mitigation
• Abstract summary: Vision-Language Models (VLMs) are frequently undermined by object hallucination.<n>We introduce Positive-and-Negative Decoding (PND), a training-free inference framework.<n>PND intervenes directly in the decoding process to enforce visual fidelity.
• Score: 31.028607494171336
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-Language Models (VLMs) are frequently undermined by object hallucination--generating content that contradicts visual reality--due to an over-reliance on linguistic priors. We introduce Positive-and-Negative Decoding (PND), a training-free inference framework that intervenes directly in the decoding process to enforce visual fidelity. PND is motivated by our key finding of a critical attention deficit in VLMs, where visual features are empirically under-weighted. Our framework corrects this via a dual-path contrast: The positive path amplifies salient visual evidence using multi-layer attention to encourage faithful descriptions, directly counteracting the attention deficit. Simultaneously, the negative path identifies and degrades the core object's features to create a strong counterfactual, which penalizes ungrounded, prior-dominant generation. By contrasting the model's outputs from these two perspectives at each step, PND steers generation towards text that is not just linguistically probable, but visually factual. Extensive experiments on benchmarks like POPE, MME, and CHAIR show that PND achieves state-of-the-art performance with up to 6.5% accuracy improvement, substantially reducing object hallucination while also enhancing descriptive detail--all without requiring any model retraining. The method generalizes effectively across diverse VLM architectures including LLaVA, InstructBLIP, InternVL, and Qwen-VL.

Related papers

• CofactVLA: Deconfounding Vision-Language-Action Models via Counterfactual Intervention [48.20065816234311]
  Vision-Language-Action (VLA) models have driven significant progress in robotic manipulation, yet they struggle with the vision-override phenomenon.<n>We propose CofactVLA, a novel causal intervention framework.<n>We show that CofactVLA establishes a new state-of-the-art across diverse simulation benchmarks.
  arXiv  Detail & Related papers  (2026-08-05T02:58:41Z)
• Diagnosing Visual Ignorance in Vision-Language Models [29.2851901986069]
  Vision-Language Models (VLMs) frequently rely on language priors, producing confident answers that are weakly grounded in visual evidence.<n>In this work, we study language-prior reliance from both mechanistic and behavioral perspectives.<n>We find that language-prior reliance is a systematic routing failure affecting both model internals and benchmark validity.
  arXiv  Detail & Related papers  (2026-06-05T04:16:01Z)
• Breaking the Illusion: When Positive Meets Negative in Multimodal Decoding [30.74548183895207]
  We introduce Positive-and-Negative Decoding (PND), a training-free inference framework.<n>PND intervenes directly in the decoding process to enforce visual fidelity.<n> Experiments on POPE, MME, and CHAIR demonstrate state-of-the-art performance without retraining.
  arXiv  Detail & Related papers  (2026-04-22T15:11:41Z)
• See Fair, Speak Truth: Equitable Attention Improves Grounding and Reduces Hallucination in Vision-Language Alignment [16.616065291567445]
  Multimodal large language models (MLLMs) frequently hallucinate objects that are absent from the visual input.<n>We propose DOP-OBC, a training-free and architecture-agnostic decoding strategy built on the principle of equitable attention.
  arXiv  Detail & Related papers  (2026-04-10T11:01:48Z)
• Predictive Regularization Against Visual Representation Degradation in Multimodal Large Language Models [84.94288033791346]
  We conduct a detailed diagnostic analysis to unveil a pervasive issue: visual representation degradation in MLLMs.<n>We attribute this phenomenon to a visual sacrifice driven by the singular text-generation objective, where the model compromises its visual fidelity to optimize for answer generation.<n>We propose Predictive Regularization to force degraded intermediate features to predict initial visual features, thereby maintaining the inherent visual attributes of the MLLM's internal representations.
  arXiv  Detail & Related papers  (2026-03-21T13:10:37Z)
• ReViP: Reducing False Completion in Vision-Language-Action Models with Vision-Proprioception Rebalance [50.05984919728878]
  We present ReViP, a novel VLA framework with Vision-Proprioception Rebalance to enhance visual grounding and robustness under perturbations.<n>Specifically, we use an external VLM as a task-stage observer to extract real-time task-centric visual cues from visual observations.<n>To evaluate false completion, we propose the first False-Completion Benchmark Suite built on LIBERO with controlled settings such as Object-Drop.
  arXiv  Detail & Related papers  (2026-01-23T11:31:07Z)
• Attention-space Contrastive Guidance for Efficient Hallucination Mitigation in LVLMs [9.043999205886658]
  Hallucinations in large vision-language models often arise when language priors dominate over visual evidence.<n>We propose Contrastive Guidance (ACG), a single-pass mechanism that operates within self-attention layers to construct both vision-language and language-only attention paths.<n>ACG achieves state-of-the-art faithfulness and caption quality while significantly reducing computational cost.
  arXiv  Detail & Related papers  (2026-01-20T08:04:18Z)
• Seeing to Act, Prompting to Specify: A Bayesian Factorization of Vision Language Action Policy [59.44168425139687]
  BayesVLA is a Bayesian factorization that decomposes the policy into a visual-action prior, supporting seeing-to-act, and a language-conditioned likelihood, enabling prompt-to-specify.<n>Experiments show superior generalization to unseen instructions, objects, and environments compared to existing methods.
  arXiv  Detail & Related papers  (2025-12-12T01:59:23Z)
• Mitigating Multimodal Hallucinations via Gradient-based Self-Reflection [49.26064449816502]
  We propose a Gradient-based Influence-Aware Constrained Decoding (GACD) method to address text-visual bias and co-occurrence bias.<n>GACD effectively reduces hallucinations and improves the visual grounding of MLLM outputs.
  arXiv  Detail & Related papers  (2025-09-03T08:13:52Z)
• ViCrit: A Verifiable Reinforcement Learning Proxy Task for Visual Perception in VLMs [98.27348724529257]
  We introduce ViCrit (Visual Caption Hallucination Critic), an RL proxy task that trains VLMs to localize a subtle, synthetic visual hallucination injected into paragraphs of human-written image captions.<n>Models trained with the ViCrit Task exhibit substantial gains across a variety of vision-language models benchmarks.
  arXiv  Detail & Related papers  (2025-06-11T19:16:54Z)
• OViP: Online Vision-Language Preference Learning for VLM Hallucination [44.14029765850719]
  Large vision-language models (LVLMs) remain vulnerable to hallucination, often generating content misaligned with visual inputs.<n>We propose an Online Vision-language Preference Learning framework that dynamically constructs contrastive training data based on the model's own hallucinations.
  arXiv  Detail & Related papers  (2025-05-21T19:26:09Z)
• Treble Counterfactual VLMs: A Causal Approach to Hallucination [6.3952983618258665]
  VisionLanguage Models (VLMs) have advanced multi-modal tasks like image captioning, visual question answering, and reasoning.<n>They often generate hallucinated outputs inconsistent with the visual context or prompt.<n>Existing studies link hallucination to statistical biases, language priors, and biased feature learning but lack a structured causal understanding.
  arXiv  Detail & Related papers  (2025-03-08T11:13:05Z)
• Debiasing Multimodal Large Language Models via Penalization of Language Priors [38.97645845493758]
  Multimodal Large Language Models (MLLMs) have become indispensable tools in computer vision and natural language processing.<n>Despite their advancements, our investigation reveals a noteworthy bias: the generated content is often driven more by the inherent priors of the underlying Large Language Models (LLMs) than by the input image.<n>We propose two simple, training-free strategies to rectify these biases and redirect the model's focus toward visual information.
  arXiv  Detail & Related papers  (2024-03-08T12:35:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.