Breaking the Illusion: When Positive Meets Negative in Multimodal Decoding

• URL: http://arxiv.org/abs/2605.06679v1
• Date: Wed, 22 Apr 2026 15:11:41 GMT
• Title: Breaking the Illusion: When Positive Meets Negative in Multimodal Decoding
• Abstract summary: We introduce Positive-and-Negative Decoding (PND), a training-free inference framework.<n>PND intervenes directly in the decoding process to enforce visual fidelity.<n> Experiments on POPE, MME, and CHAIR demonstrate state-of-the-art performance without retraining.
• Score: 30.74548183895207
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-Language Models (VLMs) are frequently undermined by object hallucination, generating content that contradicts visual reality, due to an over-reliance on linguistic priors. We introduce Positive-and-Negative Decoding (PND), a training-free inference framework that intervenes directly in the decoding process to enforce visual fidelity. PND is motivated by our finding of an attention imbalance in VLMs, where visual features are under-weighted. Our framework introduces a dual-path contrast: a positive path that amplifies visual evidence and a negative path that constructs counterfactuals to penalize prior-dominant generation. By contrasting outputs from both paths during decoding, PND steers generation toward visually grounded results. Experiments on POPE, MME, and CHAIR demonstrate state-of-the-art performance without retraining.

Related papers

• What Do Hallucinations Reveal About Multimodal Reasoning? Diagnosing Visual Grounding Failures via Contrastive Decoding Probes [1.9916820095972823]
  We introduce SAFE, a training-free decoding framework that contrasts visually-grounded and vision-ablated generation paths.<n>This signal serves dual roles: as a practical proxy for detecting visually-ungrounded tokens, and as the basis for decoding-time penalties.<n>We argue that designing contrastive probes exemplifies a broader mission: using models as instruments for scientific understanding.
  arXiv  Detail & Related papers  (2026-09-15T05:07:01Z)
• Bridging Physical Reasoning and Task Generalization via Visual Action Outcome Reasoning Alignment [67.2775785848366]
  Two key failure modes are prominent: hallucinated chain-of-thought (CoT) reasoning that contradicts physical reality, and misalignment between the model's reasoning and actions.<n>We present VAORA, a novel reward design that directly addresses both issues.<n>VAORA introduces two complementary rewards: Visual Alignment Reward, which anchors VLM reasoning to the visual context independent of the agent action itself, and Visual-Action Alignment Reward, which grounds reasoning in the visual outcome induced by the model's action.
  arXiv  Detail & Related papers  (2026-07-07T17:27:59Z)
• Instruction-Evidence Contrastive Dual-Stream Decoding for Grounded Vision-Language Reasoning [3.944710336786154]
  Vision-Language Models (VLMs) exhibit strong performance in instruction following and open-ended vision-language reasoning.<n>They frequently generate fluent outputs that are weakly grounded in visual evidence.<n>We propose a decoding framework that balances linguistic informativeness and visual faithfulness during generation.
  arXiv  Detail & Related papers  (2026-04-28T16:18:31Z)
• Prefill-Time Intervention for Mitigating Hallucination in Large Vision-Language Models [14.32953618311784]
  Prefill-Time Intervention (PTI) is a novel steering paradigm that intervenes only once during the prefill stage.<n>PTI is modality-aware, deriving distinct directions for visual and textual representations.
  arXiv  Detail & Related papers  (2026-04-28T13:42:27Z)
• Global Context or Local Detail? Adaptive Visual Grounding for Hallucination Mitigation [31.028607494171336]
  Vision-Language Models (VLMs) are frequently undermined by object hallucination.<n>We introduce Positive-and-Negative Decoding (PND), a training-free inference framework.<n>PND intervenes directly in the decoding process to enforce visual fidelity.
  arXiv  Detail & Related papers  (2026-04-27T12:23:00Z)
• ReViP: Reducing False Completion in Vision-Language-Action Models with Vision-Proprioception Rebalance [50.05984919728878]
  We present ReViP, a novel VLA framework with Vision-Proprioception Rebalance to enhance visual grounding and robustness under perturbations.<n>Specifically, we use an external VLM as a task-stage observer to extract real-time task-centric visual cues from visual observations.<n>To evaluate false completion, we propose the first False-Completion Benchmark Suite built on LIBERO with controlled settings such as Object-Drop.
  arXiv  Detail & Related papers  (2026-01-23T11:31:07Z)
• Mitigating Multimodal Hallucinations via Gradient-based Self-Reflection [49.26064449816502]
  We propose a Gradient-based Influence-Aware Constrained Decoding (GACD) method to address text-visual bias and co-occurrence bias.<n>GACD effectively reduces hallucinations and improves the visual grounding of MLLM outputs.
  arXiv  Detail & Related papers  (2025-09-03T08:13:52Z)
• Self-Rewarding Vision-Language Model via Reasoning Decomposition [49.784411666601905]
  Vision-Language Models (VLMs) often suffer from visual hallucinations, saying things that are not actually in the image, and language shortcuts.<n>We introduce Vision-SR1, a self-rewarding method that improves visual reasoning without relying on external visual supervisions.<n>Our experiments demonstrate that Vision-SR1 improves visual reasoning, mitigates visual hallucinations, and reduces reliance on language shortcuts.
  arXiv  Detail & Related papers  (2025-08-27T08:01:03Z)
• ViCrit: A Verifiable Reinforcement Learning Proxy Task for Visual Perception in VLMs [98.27348724529257]
  We introduce ViCrit (Visual Caption Hallucination Critic), an RL proxy task that trains VLMs to localize a subtle, synthetic visual hallucination injected into paragraphs of human-written image captions.<n>Models trained with the ViCrit Task exhibit substantial gains across a variety of vision-language models benchmarks.
  arXiv  Detail & Related papers  (2025-06-11T19:16:54Z)
• OViP: Online Vision-Language Preference Learning for VLM Hallucination [44.14029765850719]
  Large vision-language models (LVLMs) remain vulnerable to hallucination, often generating content misaligned with visual inputs.<n>We propose an Online Vision-language Preference Learning framework that dynamically constructs contrastive training data based on the model's own hallucinations.
  arXiv  Detail & Related papers  (2025-05-21T19:26:09Z)
• Looking Beyond Text: Reducing Language bias in Large Vision-Language Models via Multimodal Dual-Attention and Soft-Image Guidance [67.26434607115392]
  Large vision-language models (LVLMs) have achieved impressive results in various vision-language tasks. LVLMs suffer from hallucinations caused by language bias, leading to diminished focus on images and ineffective visual comprehension. We propose LACING to address the language bias of LVLMs with muLtimodal duAl-attention meChanIsm (MDA) aNd soft-image Guidance (IFG)
  arXiv  Detail & Related papers  (2024-11-21T16:33:30Z)
• Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding [125.05295513481035]
  We introduce Visual Contrastive Decoding (VCD), a simple and training-free method that contrasts output distributions derived from original and distorted visual inputs. The proposed VCD effectively reduces the over-reliance on statistical bias and unimodal priors, two essential causes of object hallucinations. Our experiments show that VCD, without either additional training or the usage of external tools, significantly mitigates the object hallucination issue across different LVLM families.
  arXiv  Detail & Related papers  (2023-11-28T16:26:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.