Kestrel: Grounding Self-Refinement for LVLM Hallucination Mitigation

• URL: http://arxiv.org/abs/2603.16664v1
• Date: Tue, 17 Mar 2026 15:30:47 GMT
• Title: Kestrel: Grounding Self-Refinement for LVLM Hallucination Mitigation
• Abstract summary: Large vision-language models (LVLMs) have become increasingly strong but remain prone to hallucinations in multimodal tasks.<n>As training these LVLMs to avoid hallucinations becomes prohibitively expensive for larger models, training-free methods offer a cheap and flexible solution to this problem.<n>We propose Kestrel, a training-free framework for LVLM hallucination mitigation that combines an explicit visual-grounding agent with evidence-verified self-refinement mechanism.
• Score: 86.37623966653688
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large vision-language models (LVLMs) have become increasingly strong but remain prone to hallucinations in multimodal tasks, which significantly narrows their deployment. As training these LVLMs to avoid hallucinations becomes prohibitively expensive for larger models, training-free methods offer a cheap and flexible solution to this problem, yet existing approaches based on decoding or tool use often bring limited gains and/or weak interpretability. We propose Kestrel, a training-free framework for LVLM hallucination mitigation that combines an explicit visual-grounding agent with evidence-verified self-refinement mechanism. In detail, Kestrel first collects explicit visual evidence and converts tool outputs into reusable and structured textual evidence. Second, to take full advantage of these evidence, Kestrel verifies them via an LVLM judge for evidence checking, then iteratively self-refine answers based on verified evidence to reduce the risk of over-correction. Extensive experiments show that Kestrel improves performance over strong baselines across hallucination benchmarks (e.g., average +3.31% on POPE and +28.34 on MME-Hallucination with Qwen3-VL), while providing transparent verification traces for hallucination diagnosis and analysis -- e.g., both the integrated self-refinement module and grounding agent contributing an average +2.0% gain on POPE.

Related papers

• Adversarial Orthogonal Disentanglement for LVLM Hallucination Mitigation [35.94927596542353]
  Large Vision-Language Models (LVLMs) have advanced multimodal understanding, yet their reliability is limited by hallucinations.<n>We propose Adrial Orthogonal Disentanglement (AOD), a latent geometric framework for mitigating hallucinations.
  arXiv  Detail & Related papers  (2026-05-25T03:05:04Z)
• Mitigating Multimodal Hallucination via Phase-wise Self-reward [37.441718699530526]
  We introduce a new self-rewarding framework, enabling dynamic hallucination mitigation at inference time without external supervision.<n>textbfPSRD (textbfPhase-wise textbfSelf-textbfReward textbfDecoding) for online hallucination correction guided by phase-wise self-reward signals.
  arXiv  Detail & Related papers  (2026-04-20T09:04:49Z)
• HIME: Mitigating Object Hallucinations in LVLMs via Hallucination Insensitivity Model Editing [6.021803204524807]
  Large Vision-Language Models (LVLMs) have demonstrated impressive multimodal understanding capabilities.<n>LVLMs are prone to object hallucination, where models describe non-existent objects or attribute incorrect factual information.<n>We propose Hallucination Insensitivity Model Editing (HIME), a layer-adaptive weight editing approach that selectively modifies latent features to suppress hallucinations.
  arXiv  Detail & Related papers  (2026-02-21T04:16:17Z)
• PruneHal: Reducing Hallucinations in Multi-modal Large Language Models through Adaptive KV Cache Pruning [87.35309934860938]
  hallucinations in large language models (MLLMs) are strongly associated with insufficient attention allocated to visual tokens.<n>We propose textbfPruneHal, a training-free, simple yet effective method that leverages adaptive KV cache pruning to enhance the model's focus on critical visual information.
  arXiv  Detail & Related papers  (2025-10-22T02:41:07Z)
• Mitigating Hallucinations in Large Vision-Language Models by Self-Injecting Hallucinations [73.37711261605271]
  hallucination mitigation methods are mainly based on preference alignment and require external human annotations or auxiliary models for preference data collection.<n>We propose Autonomous Preference Alignment via Self-Injection (APASI), a novel and generalizable method that mitigates hallucinations without external dependencies.<n>APASI leverages the target LVLM to self-inject hallucinations into a generated response, creating a pair of responses with varying preference levels.
  arXiv  Detail & Related papers  (2025-09-14T14:26:53Z)
• Identify, Isolate, and Purge: Mitigating Hallucinations in LVLMs via Self-Evolving Distillation [52.52962914918779]
  hallucination issues significantly limit their credibility and application potential.<n>Existing mitigation methods rely on external tools or the comparison of multi-round inference.<n>We propose textbfSElf-textbfEvolving textbfDistillation (textbfSEED), which identifies hallucinations within the inner knowledge of LVLMs, isolates and purges them, and then distills the purified knowledge back into the model.
  arXiv  Detail & Related papers  (2025-07-07T05:56:19Z)
• MIRAGE: Assessing Hallucination in Multimodal Reasoning Chains of MLLM [58.2298313720146]
  Multimodal hallucinations are multi-sourced and arise from diverse causes.<n>Existing benchmarks fail to adequately distinguish between perception-induced hallucinations and reasoning-induced hallucinations.
  arXiv  Detail & Related papers  (2025-05-30T05:54:36Z)
• Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resampling [78.78822033285938]
  Vision-Language Models (VLMs) excel at visual understanding but often suffer from visual hallucinations.<n>In this work, we introduce REVERSE, a unified framework that integrates hallucination-aware training with on-the-fly self-verification.
  arXiv  Detail & Related papers  (2025-04-17T17:59:22Z)
• Efficient Contrastive Decoding with Probabilistic Hallucination Detection - Mitigating Hallucinations in Large Vision Language Models - [1.2499537119440245]
  Efficient Contrastive Decoding (ECD) is a simple method that leverages probabilistic hallucination detection to shift the output distribution towards contextually accurate answers at inference time.<n>Our experiments show that ECD effectively mitigates hallucinations, outperforming state-of-the-art methods with respect to performance on LVLM benchmarks and computation time.
  arXiv  Detail & Related papers  (2025-04-16T14:50:25Z)
• Attention-guided Self-reflection for Zero-shot Hallucination Detection in Large Language Models [41.979015935353814]
  We introduce a novel Attention-Guided SElf-Reflection (AGSER) approach for zero-shot hallucination detection in Large Language Models (LLMs)<n>The AGSER method utilizes attention contributions to categorize the input query into attentive and non-attentive queries.<n>In addition to its efficacy in detecting hallucinations, AGSER notably reduces computational overhead, requiring only three passes through the LLM and utilizing two sets of tokens.
  arXiv  Detail & Related papers  (2025-01-17T07:30:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.