Mitigating Action-Relation Hallucinations in LVLMs via Relation-aware Visual Enhancement

• URL: http://arxiv.org/abs/2605.11808v1
• Date: Tue, 12 May 2026 09:03:19 GMT
• Title: Mitigating Action-Relation Hallucinations in LVLMs via Relation-aware Visual Enhancement
• Abstract summary: Large Vision-Language Models (LVLMs) have achieved remarkable performance on diverse vision-language tasks.<n>LVLMs still suffer from hallucinations, generating text that contradicts the visual input.<n>We propose a framework to locate action-relevant image regions and enhance the LVLM's attention to those regions.
• Score: 8.424418427339337
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Vision-Language Models (LVLMs) have achieved remarkable performance on diverse vision-language tasks. However, LVLMs still suffer from hallucinations, generating text that contradicts the visual input. Existing research has primarily focused on mitigating object hallucinations, but often overlooks more complex relation hallucinations, particularly action relations involving interactions between objects. In this study, we empirically observe that the primary cause of action-relation hallucinations in LVLMs is the insufficient attention allocated to visual information. Thus, we propose a framework to locate action-relevant image regions and enhance the LVLM's attention to those regions. Specifically, we define the Action-Relation Sensitivity (ARS) score to identify attention heads that are most sensitive to action-relation changes, thereby localizing action-relevant image regions that contain key visual cues. Then, we propose the Relation-aware Visual Enhancement (RVE) method to enhance the LVLM's attention to these action-relevant image regions. Extensive experiments demonstrate that, compared to existing baselines, our method achieves superior performance in mitigating action-relation hallucinations with negligible additional inference cost. Furthermore, it effectively generalizes to spatial-relation hallucinations and object hallucinations.

Related papers

• Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucination [23.128973540926552]
  We find that both real and hallucinated objects receive equally strong visual attention in the model's mid-to-late layers.<n>We propose a training-free Detect-Mitigate framework comprising a Logit-Lens Consistency Check to detect hallucination.
  arXiv  Detail & Related papers  (2026-08-07T14:56:12Z)
• When Prompts Override Vision: Prompt-Induced Hallucinations in LVLMs [54.411658510110215]
  We propose HalluScope to better understand the extent to which different factors induce hallucinations.<n>HalluVL-DPO is a framework for fine-tuning off-the-shelf LVLMs towards more visually grounded responses.
  arXiv  Detail & Related papers  (2026-04-23T17:54:36Z)
• R-CoV: Region-Aware Chain-of-Verification for Alleviating Object Hallucinations in LVLMs [88.62912181680413]
  Region-aware Chain-of-Verification (R-CoV) is a visual chain-of-verification method to alleviate object hallucinations in large vision-language models.<n>R-CoV consists of six steps: initial response generation, entity extraction, coordinate generation, region description, verification execution, and final response generation.
  arXiv  Detail & Related papers  (2026-04-22T15:41:33Z)
• Segmentation-Based Attention Entropy: Detecting and Mitigating Object Hallucinations in Large Vision-Language Models [9.388076929154673]
  Large Vision-Language Models (LVLMs) achieve strong performance on many multimodal tasks, but object hallucinations severely undermine their reliability.<n>Most existing studies focus on the text modality, attributing hallucinations to overly strong language priors and insufficient visual grounding.<n>We propose Attention Entropy (SAE), which leverages semantic segmentation to quantify visual attention uncertainty in an object-level semantic space.
  arXiv  Detail & Related papers  (2026-03-17T14:19:22Z)
• IKOD: Mitigating Visual Attention Degradation in Large Vision-Language Models [20.036659182106806]
  We show that Large Vision-Language Models (LVLMs) exhibit a long-term bias where hallucinations increase as the sequence length grows.<n>We propose Image attention-guided Key-value merging cOllaborative Decoding (IKOD), a collaborative decoding strategy generating more image-focused sequences.
  arXiv  Detail & Related papers  (2025-08-05T14:05:15Z)
• CAI: Caption-Sensitive Attention Intervention for Mitigating Object Hallucination in Large Vision-Language Models [60.0300765815417]
  Large Vision-Language Models (LVLMs) frequently produce content that deviates from visual information, leading to object hallucination.<n>We propose Caption-sensitive Attention Intervention (CAI), a training-free, plug-and-play hallucination mitigation method.
  arXiv  Detail & Related papers  (2025-06-30T07:52:36Z)
• Mitigating Hallucination for Large Vision Language Model by Inter-Modality Correlation Calibration Decoding [66.06337890279839]
  Large vision-language models (LVLMs) have shown remarkable capabilities in visual-language understanding for downstream multi-modal tasks.<n>LVLMs still suffer from generating hallucinations in complex generation tasks, leading to inconsistencies between visual inputs and generated content.<n>We propose an Inter-Modality Correlation Decoding (IMCCD) method to mitigate hallucinations in LVLMs in a training-free manner.
  arXiv  Detail & Related papers  (2025-01-03T17:56:28Z)
• Cracking the Code of Hallucination in LVLMs with Vision-aware Head Divergence [69.86946427928511]
  We investigate the internal mechanisms driving hallucination in large vision-language models (LVLMs)<n>We introduce Vision-aware Head Divergence (VHD), a metric that quantifies the sensitivity of attention head outputs to visual context.<n>We propose Vision-aware Head Reinforcement (VHR), a training-free approach to mitigate hallucination by enhancing the role of vision-aware attention heads.
  arXiv  Detail & Related papers  (2024-12-18T15:29:30Z)
• Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens [7.806633929976787]
  Hallucinations in Large Vision-Language Models (LVLMs) significantly undermine their reliability.<n>This paper addresses how LVLMs process visual information and whether this process causes hallucination.<n>We propose a simple inference-time method that adjusts visual attention by integrating information across various heads.
  arXiv  Detail & Related papers  (2024-11-23T03:40:05Z)
• Evaluating and Analyzing Relationship Hallucinations in Large Vision-Language Models [69.79709804046325]
  We introduce R-Bench, a novel benchmark for evaluating Vision Relationship Hallucination. R-Bench features image-level questions that focus on the existence of relationships and instance-level questions that assess local visual comprehension. We identify three types of relationship co-occurrences that lead to hallucinations: relationship-relationship, subject-relationship, and relationship-object.
  arXiv  Detail & Related papers  (2024-06-24T08:42:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.