AdaVBoost: Mitigating Hallucinations in LVLMs via Token-Level Adaptive Visual Attention Boosting

• URL: http://arxiv.org/abs/2602.13600v1
• Date: Sat, 14 Feb 2026 04:44:37 GMT
• Title: AdaVBoost: Mitigating Hallucinations in LVLMs via Token-Level Adaptive Visual Attention Boosting
• Authors: Jiacheng Zhang, Feng Liu, Chao Du, Tianyu Pang,
• Abstract summary: Visual attention boosting has emerged as a promising direction for mitigating hallucinations in Large Vision-Language Models (LVLMs)<n>We propose AdaVBoost, a token-level visual attention boosting framework that adaptively determines how much attention to boost at each generation step.<n>We show that AdaVBoost significantly outperforms baseline methods across multiple LVLMs and hallucination benchmarks.
• Score: 58.47431497789902
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Visual attention boosting has emerged as a promising direction for mitigating hallucinations in Large Vision-Language Models (LVLMs), where existing methods primarily focus on where to boost by applying a predefined scaling to the attention of method-specific visual tokens during autoregressive generation. In this paper, we identify a fundamental trade-off in these methods: a predefined scaling factor can be too weak at some generation steps, leaving hallucinations unresolved, yet too strong at others, leading to new hallucinations. Motivated by this finding, we propose AdaVBoost, a token-level visual attention boosting framework that adaptively determines how much attention to boost at each generation step. Specifically, we introduce Visual Grounding Entropy (VGE) to estimate hallucination risk, which leverages visual grounding as a complementary signal to capture evidence mismatches beyond entropy. Guided by VGE, AdaVBoost applies stronger visual attention boosting to high-risk tokens and weaker boosting to low-risk tokens, enabling token-level adaptive intervention at each generation step. Extensive experiments show that AdaVBoost significantly outperforms baseline methods across multiple LVLMs and hallucination benchmarks.

Related papers

• Revealing and Enhancing Core Visual Regions: Harnessing Internal Attention Dynamics for Hallucination Mitigation in LVLMs [67.69730908817321]
  Internal Positive Attention Dynamics (PAD) in LVLMs naturally reveal semantically core visual regions under the distortions of attention sinks.<n>We propose Positive Attention Dynamics Enhancement (PADE), a training-free attention intervention that constructs a PAD map to identify semantically core visual regions.
  arXiv  Detail & Related papers  (2026-02-17T13:08:06Z)
• Attention to details, logits to truth: visual-aware attention and logits enhancement to mitigate hallucinations in LVLMs [12.578567672069601]
  We propose a training free attentional intervention algorithm to enhance the attention of task-relevant tokens.<n>To enhance the contribution of visual tokens, we inject visual attention values into the beam search decoding to identify solutions with higher visual attention.
  arXiv  Detail & Related papers  (2026-02-10T08:26:50Z)
• Hallucination Begins Where Saliency Drops [18.189047289404325]
  hallucinations frequently arise when preceding output tokens exhibit low saliency toward the prediction of the next token.<n>We introduce LVLMs-Saliency, a gradient-aware diagnostic framework that quantifies the visual grounding strength of each output token.<n>Our method significantly reduces hallucination rates while preserving fluency and task performance, offering a robust and interpretable solution.
  arXiv  Detail & Related papers  (2026-01-28T05:50:52Z)
• Visual Multi-Agent System: Mitigating Hallucination Snowballing via Visual Flow [99.54291580187817]
  Multi-Agent System (MAS) powered by Visual Language Models (VLMs) enables challenging tasks but suffers from a novel failure term, visual hallucination snowballing.<n>We provide detailed insights into the essence of hallucination snowballing regarding the reduction of visual attention allocation.<n>We propose ViF, a lightweight, plug-and-play mitigation paradigm that relays inter-agent messages with Visual Flow powered by the selected visual relay tokens and applies attention reallocation to amplify this pattern.
  arXiv  Detail & Related papers  (2025-09-26T02:43:24Z)
• CAI: Caption-Sensitive Attention Intervention for Mitigating Object Hallucination in Large Vision-Language Models [60.0300765815417]
  Large Vision-Language Models (LVLMs) frequently produce content that deviates from visual information, leading to object hallucination.<n>We propose Caption-sensitive Attention Intervention (CAI), a training-free, plug-and-play hallucination mitigation method.
  arXiv  Detail & Related papers  (2025-06-30T07:52:36Z)
• Attention Hijackers: Detect and Disentangle Attention Hijacking in LVLMs for Hallucination Mitigation [123.54980913741828]
  Large Vision-Language Models (LVLMs) remain vulnerable to hallucinations.<n>We propose a novel, training-free strategy namely Attention HIjackers Detection and Disentanglement (AID)<n>AID identifies Attention Hijackers by calculating instruction-driven visual salience.<n>Next, Attention Disentanglement mechanism is proposed to mask the visual attention of these identified Hijackers.<n>Re-Disentanglement recalculates the balance between instruction-driven and image-driven visual salience to avoid over-masking effects.
  arXiv  Detail & Related papers  (2025-03-11T09:35:55Z)
• MINT: Mitigating Hallucinations in Large Vision-Language Models via Token Reduction [6.416957959150438]
  Hallucinations hinder the application of Large Vision-Language Models (LVLMs) in domains that require high reliability.<n>We propose MINT, a training-free decoding strategy, MItigating hallucinations via tokeN reducTion.<n>Our approach achieves a 4% improvement in mitigating hallucinations caused by distracted perception compared to original models.
  arXiv  Detail & Related papers  (2025-02-02T08:34:57Z)
• Cracking the Code of Hallucination in LVLMs with Vision-aware Head Divergence [69.86946427928511]
  We investigate the internal mechanisms driving hallucination in large vision-language models (LVLMs)<n>We introduce Vision-aware Head Divergence (VHD), a metric that quantifies the sensitivity of attention head outputs to visual context.<n>We propose Vision-aware Head Reinforcement (VHR), a training-free approach to mitigate hallucination by enhancing the role of vision-aware attention heads.
  arXiv  Detail & Related papers  (2024-12-18T15:29:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.