Mitigating Object Hallucinations in Vision-Language Models through Region-Aware Attention Recalibration

• URL: http://arxiv.org/abs/2605.24957v1
• Date: Sun, 24 May 2026 09:14:49 GMT
• Title: Mitigating Object Hallucinations in Vision-Language Models through Region-Aware Attention Recalibration
• Abstract summary: We introduce a training-free inference strategy to dynamically correct semantic drift without relying on abrupt truncations.<n>By computing an outlier-resistant statistical midpoint across various attention heads, we establish a stable anchor for reliable visual representations.<n>This recalibration process effectively rectifies visual-semantic misalignments while fully preserving generative fluency and language priors.
• Score: 7.647944993041615
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The generation of factually incorrect objects, commonly known as object hallucination, remains a persistent challenge in Large Vision-Language Models (LVLMs). Current approaches to address this issue - ranging from expensive data-driven fine-tuning and high-latency contrastive decoding to rigid attention head truncation - frequently compromise either computational efficiency or the continuity of the model's feature space. To overcome these limitations, we introduce a novel, training-free inference strategy that operates as a region-aware adaptive weighting mechanism to dynamically correct semantic drift without relying on abrupt heuristic truncations. By computing an outlier-resistant statistical midpoint across various attention heads, we establish a stable anchor for reliable visual representations. We then utilize the inter-head disagreement mapped across regions to dynamically determine intervention budgets, gently suppressing hallucination-inducing attention paths through a continuous penalty modulation. This recalibration process effectively rectifies visual-semantic misalignments while fully preserving generative fluency and language priors. Comprehensive evaluations on standard multimodal benchmarks, including CHAIR, POPE, and MME, reveal that our strategy substantially curtails both instance- and sentence-level hallucinations. The results demonstrate state-of-the-art performance against contemporary baselines, confirming our method's efficiency and algorithmic robustness. Our code will be public.

Related papers

• See Only When Needed: Context-Aware Attention Intervention for Mitigating Hallucinations in LVLMs [81.71384150441163]
  We propose Context-aware Attention Intervention (CAI) for large vision-language models (LVLMs)<n>CAI enforces a see only when needed principle via two-axis selectivity: where to look and when to intervene.<n> experiments across multiple LVLM backbones and benchmarks show that CAI achieves state-of-the-art hallucination mitigation.
  arXiv  Detail & Related papers  (2026-06-29T06:35:47Z)
• Mitigating Hallucinations in Large Vision-Language Models via Causal Route Gating [4.514623850633406]
  Large vision-language models (LVLMs) often hallucinate content that is fluent yet unsupported by the image.<n>We show that a key failure mode arises from route competition, causing the decoder to follow linguistic priors over visual evidence.<n>We propose a training-free, decision-aligned intervention that decomposes each attention head into a visual route and a text route.
  arXiv  Detail & Related papers  (2026-05-20T13:29:29Z)
• ACT Now: Preempting LVLM Hallucinations via Adaptive Context Integration [71.21097024566285]
  Large-Language Models (LVLMs) frequently suffer from severe hallucination issues.<n>Existing mitigation strategies rely on linguistic, single-step states to enhance visual focus or suppress strong priors.<n>We propose Adaptive Context in VisionTegration (ACT), a training-free inference intervention method that mitigates hallucination through the adaptive integration of contextual information.
  arXiv  Detail & Related papers  (2026-04-01T14:49:50Z)
• Seeing to Ground: Visual Attention for Hallucination-Resilient MDLLMs [47.94507630961399]
  We introduce VISAGE, a training-free decoding framework that calibrates the objective at inference time.<n>We provide an analytical stability guarantee establishing that VISAGE maintains a bounded objective loss under estimation error.<n> Evaluations across hallucination-sensitive and general-purpose benchmarks demonstrate the robustness of the framework.
  arXiv  Detail & Related papers  (2026-03-26T17:53:49Z)
• Attention-space Contrastive Guidance for Efficient Hallucination Mitigation in LVLMs [9.043999205886658]
  Hallucinations in large vision-language models often arise when language priors dominate over visual evidence.<n>We propose Contrastive Guidance (ACG), a single-pass mechanism that operates within self-attention layers to construct both vision-language and language-only attention paths.<n>ACG achieves state-of-the-art faithfulness and caption quality while significantly reducing computational cost.
  arXiv  Detail & Related papers  (2026-01-20T08:04:18Z)
• Adaptive Residual-Update Steering for Low-Overhead Hallucination Mitigation in Large Vision Language Models [13.32858759983739]
  Large Vision-Language Models (LVLMs) often suffer from object hallucination, generating text inconsistent with visual inputs.<n>Existing inference-time interventions to mitigate this issue present a challenging trade-off.<n>We present Residual-Update Directed DEcoding Regulation (RUDDER), a framework that steers LVLMs towards visually-grounded generation.
  arXiv  Detail & Related papers  (2025-11-13T13:29:38Z)
• Mitigating Hallucination of Large Vision-Language Models via Dynamic Logits Calibration [8.192590936983347]
  Large Vision-Language Models (LVLMs) have demonstrated significant advancements in multimodal understanding.<n>They are frequently hampered by hallucination-the generation of text that contradicts visual input.<n>Existing training-free decoding strategies exhibit critical limitations.<n>This paper introduces Dynamic Logits (DLC), a novel training-free decoding framework designed to align text generation with visual evidence at inference time.
  arXiv  Detail & Related papers  (2025-06-26T17:35:40Z)
• Multimodal LLM-Guided Semantic Correction in Text-to-Image Diffusion [52.315729095824906]
  MLLM Semantic-Corrected Ping-Pong-Ahead Diffusion (PPAD) is a novel framework that introduces a Multimodal Large Language Model (MLLM) as a semantic observer during inference.<n>It performs real-time analysis on intermediate generations, identifies latent semantic inconsistencies, and translates feedback into controllable signals that actively guide the remaining denoising steps.<n>Extensive experiments demonstrate PPAD's significant improvements.
  arXiv  Detail & Related papers  (2025-05-26T14:42:35Z)
• PerturboLLaVA: Reducing Multimodal Hallucinations with Perturbative Visual Training [56.172959986096316]
  This paper aims to address the challenge of hallucinations in Multimodal Large Language Models (MLLMs)<n>HalFscore is a novel metric built upon the language graph and is designed to evaluate both the accuracy and completeness of dense captions at a granular level.<n>PerturboLLaVA significantly improves the fidelity of generated captions, outperforming existing approaches in handling multimodal hallucinations.
  arXiv  Detail & Related papers  (2025-03-09T07:07:03Z)
• Mitigating Hallucination for Large Vision Language Model by Inter-Modality Correlation Calibration Decoding [66.06337890279839]
  Large vision-language models (LVLMs) have shown remarkable capabilities in visual-language understanding for downstream multi-modal tasks.<n>LVLMs still suffer from generating hallucinations in complex generation tasks, leading to inconsistencies between visual inputs and generated content.<n>We propose an Inter-Modality Correlation Decoding (IMCCD) method to mitigate hallucinations in LVLMs in a training-free manner.
  arXiv  Detail & Related papers  (2025-01-03T17:56:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.