Dual-Pathway Circuits of Object Hallucination in Vision-Language Models

• URL: http://arxiv.org/abs/2605.13156v1
• Date: Wed, 13 May 2026 08:20:01 GMT
• Title: Dual-Pathway Circuits of Object Hallucination in Vision-Language Models
• Abstract summary: Vision-language models (VLMs) have demonstrated remarkable capabilities in bridging visual perception and natural language understanding.<n>They often produce object hallucinations, describing content absent from the input image, which limits their reliability and interpretability.<n>We propose Dual-Pathway Circuit Analysis, a framework that identifies and characterizes hallucination-related circuits inVLMs.
• Score: 8.969579206677897
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-language models (VLMs) have demonstrated remarkable capabilities in bridging visual perception and natural language understanding, enabling a wide range of multimodal reasoning tasks. However, they often produce object hallucinations, describing content absent from the input image, which limits their reliability and interpretability. To address this limitation, we propose Dual-Pathway Circuit Analysis, a framework that identifies and characterizes hallucination-related circuits in VLMs for mechanistic understanding and causal probing. We first apply activation patching across five architecturally diverse VLMs to identify a visual grounding pathway that supports correct predictions and a hallucination pathway that drives erroneous outputs. We then introduce Conditional Pathway Analysis (CPA) to characterize pathway-level interactions, revealing that grounding components remain strongly redundant in both correct and hallucinating samples but undergo a consistent polarity flip, shifting from supporting the ground truth on correct samples to aligning with the hallucinated answer on erroneous ones. We further perform targeted suppression of hallucination-pathway components, showing that scaling these components reduces object hallucination by up to 76% with minimal accuracy cost, and validate that the same circuit selectively transfers to relational but not attribute hallucination. Evaluations on POPE-adversarial and AMBER show that the identified circuits are consistent across architectures, support causal intervention, and transfer selectively across hallucination types.

Related papers

• Role-Break in Attention Heads: Understanding and Detecting Hallucinations in VLMs [49.743008490747094]
  Vision-Language Models (VLMs) remain prone to hallucinations, producing content that is inconsistent with or unsupported by the input image.<n>Under a unified head-level view, we find that hallucination-induced changes manifest as localized deviations from each head's faithful contextual behavior.<n> Detailed analysis reveals that these deviations are systematically organized across attention heads, contextual sources, and deviation directions, and that the resulting signal is linearly readable once head identity is preserved.
  arXiv  Detail & Related papers  (2026-07-31T13:33:51Z)
• HIVE: Understanding Post-Hallucination Reasoning in Vision Language Models [70.67112531913669]
  Hallucinations in vision language models are commonly treated as semantic errors.<n>Prior work mainly focuses on detecting or suppressing hallucinations at generation time.<n>We study Post Hallucination Reasoning (PHR), the stage in which hallucinated semantics enter the model's inference context.
  arXiv  Detail & Related papers  (2026-07-08T15:02:11Z)
• Detecting Clinical Hallucinations in LVLMs via Counterfactual Visual Grounding Uncertainty [22.008964207160663]
  We present a vision-traceable hallucination detection framework that audits arbitrary LVLM responses via visual evidence grounding.<n>We extract visually verifiable entities and use a medical-domain-adapted Qwen-VL grounding verifier to localize each entity on the input image.<n>Experiments on multiple medical imaging modalities and LVLM backbones demonstrate that our method consistently improves hallucination detection performance.
  arXiv  Detail & Related papers  (2026-06-26T18:15:38Z)
• Rethinking Visual Neglect: Steering via Context-Preference for MLLM Hallucination Mitigation [5.041079621345155]
  We argue that the image, as a context, simultaneously competes with the model's parametric knowledge and the textual context.<n>We propose a training-free framework, Context-Preference Steering (CAS), which extracts two semantically distinct Context Preference Vectors.<n>Experiments show that CAS substantially mitigates object hallucinations without increasing residual latency and preserves native text-generation quality.
  arXiv  Detail & Related papers  (2026-05-27T05:33:06Z)
• When Prompts Override Vision: Prompt-Induced Hallucinations in LVLMs [54.411658510110215]
  We propose HalluScope to better understand the extent to which different factors induce hallucinations.<n>HalluVL-DPO is a framework for fine-tuning off-the-shelf LVLMs towards more visually grounded responses.
  arXiv  Detail & Related papers  (2026-04-23T17:54:36Z)
• Mitigating Entangled Steering in Large Vision-Language Models for Hallucination Reduction [49.96701537295129]
  Large Vision-Language Models (LVLMs) have achieved remarkable success across cross-modal tasks but remain hindered by hallucinations.<n>Existing methods mitigate hallucinations but often alter generation behavior, resulting in shorter outputs and shifted token distributions.<n>We propose MESA, an effective plug-and-play framework that performs controlled and selective latent intervention for hallucination mitigation.
  arXiv  Detail & Related papers  (2026-04-09T07:31:27Z)
• Intervene-All-Paths: Unified Mitigation of LVLM Hallucinations across Alignment Formats [36.4775746930584]
  We propose a comprehensive intervention framework aligned with the transformer's causal architecture in Large Vision-Language Models (LVLMs)<n>We find that hallucinations in LVLMs do not arise from a single causal path, but from the interplay among image-to-input-text, image-to-output-text, and text-to-text pathways.<n>Building on these insights, we propose simple yet effective methods to identify and intervene on critical hallucination heads within each pathway, tailored to discriminative and generative formats.
  arXiv  Detail & Related papers  (2025-11-21T13:57:38Z)
• Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs [26.144870818163387]
  We propose a framework that models hallucination process via a structural causal graph.<n>We introduce VTACR, a novel metric that quantifies the modality contribution imbalance during decoding.<n>We design a fine-language attention intervention mechanism that dynamically adjusts token- and layer-wise attention.
  arXiv  Detail & Related papers  (2025-11-12T06:13:26Z)
• Review of Hallucination Understanding in Large Language and Vision Models [65.29139004945712]
  We present a framework for characterizing both image and text hallucinations across diverse applications.<n>Our investigations reveal that hallucinations often stem from predictable patterns in data distributions and inherited biases.<n>This survey provides a foundation for developing more robust and effective solutions to hallucinations in real-world generative AI systems.
  arXiv  Detail & Related papers  (2025-09-26T09:23:08Z)
• Exploring and Mitigating Fawning Hallucinations in Large Language Models [52.444712272909435]
  We analyze fawning hallucinations in various natural language processing tasks.<n>We tailor the so-termed contrastive decoding method for fawning-hallucination mitigation.
  arXiv  Detail & Related papers  (2025-08-31T14:29:54Z)
• Why and How LLMs Hallucinate: Connecting the Dots with Subsequence Associations [82.42811602081692]
  This paper introduces a subsequence association framework to systematically trace and understand hallucinations.<n>Key insight is hallucinations that arise when dominant hallucinatory associations outweigh faithful ones.<n>We propose a tracing algorithm that identifies causal subsequences by analyzing hallucination probabilities across randomized input contexts.
  arXiv  Detail & Related papers  (2025-04-17T06:34:45Z)
• Treble Counterfactual VLMs: A Causal Approach to Hallucination [6.3952983618258665]
  VisionLanguage Models (VLMs) have advanced multi-modal tasks like image captioning, visual question answering, and reasoning.<n>They often generate hallucinated outputs inconsistent with the visual context or prompt.<n>Existing studies link hallucination to statistical biases, language priors, and biased feature learning but lack a structured causal understanding.
  arXiv  Detail & Related papers  (2025-03-08T11:13:05Z)
• Mitigating Hallucinations in Large Vision-Language Models by Adaptively Constraining Information Flow [32.039946174953236]
  Large vision-language models show tremendous potential in understanding visual information through human languages.<n>They are prone to suffer from object hallucination, i.e., the generated image descriptions contain objects that do not exist in the image.<n>We propose Variational Information Bottleneck (VIB) to alleviate overconfidence by introducing hallucination noise.
  arXiv  Detail & Related papers  (2025-02-28T05:56:23Z)
• Towards a Systematic Evaluation of Hallucinations in Large-Vision Language Models [57.58426038241812]
  Large Vision-Language Models (LVLMs) have demonstrated remarkable performance in complex multimodal tasks.<n>These models still suffer from hallucinations when required to implicitly recognize or infer diverse visual entities from images.<n>We propose a novel visual question answering (VQA) benchmark that employs contextual reasoning prompts as hallucination attacks.
  arXiv  Detail & Related papers  (2024-12-29T23:56:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.