YARD: Y-Architecture Register Decoding for Efficient Hallucination Mitigation in Large Vision-Language Models

• URL: http://arxiv.org/abs/2605.31429v1
• Date: Fri, 29 May 2026 15:23:39 GMT
• Title: YARD: Y-Architecture Register Decoding for Efficient Hallucination Mitigation in Large Vision-Language Models
• Abstract summary: Contrastive decoding (CD) seeks to mitigate hallucinations in Large Vision-Language Models (LVLMs) by contrasting the output distributions of a standard model and a visually degraded model.<n>Existing training-free CD methods suffer from sub-optimal degraded branches.<n>We propose YARD, a training-free Y-Architecture Register Decoding framework.
• Score: 16.521294712284813
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Contrastive decoding (CD) seeks to mitigate hallucinations in Large Vision-Language Models (LVLMs) by contrasting the output distributions of a standard model and a visually degraded model. However, existing training-free CD methods suffer from sub-optimal degraded branches: completely dropping visual tokens is too extreme and induces language hallucinations, while corrupting input images offers coarse control over visual evidence and suffers from high inference latency due to requiring two full forward passes. To address these dilemmas, we propose YARD, a training-free Y-Architecture Register Decoding framework. Motivated by the observation that reliable text-to-vision grounding predominantly emerges in the middle decoder layers, YARD constructs the degraded branch internally by sharing shallow-layer computations and branching exactly at this critical stage. For the degraded branch, YARD replaces patch-level visual tokens with register tokens, which preserve global image semantics but lack fine-grained local evidence. This image-aware yet locally under-grounded design provides a faithful contrastive signal without extreme modality mismatch, while the Y-architecture strictly avoids a costly second forward pass. Extensive experiments on generative and discriminative hallucination benchmarks demonstrate that YARD consistently achieves state-of-the-art hallucination mitigation across multiple LVLMs, alongside a significant reduction in inference latency.

Related papers

• UniProbe: A Learnable Token-Level Hallucination Detector for Large VLMs using Multi-Structural Internal Representations [61.85916280854257]
  Large Vision-Language Models (LVLMs) achieve impressive visual reasoning and dialogue capabilities, yet frequently hallucinate content unsupported by the visual input.<n>We introduce textbfUniProbe, a lightweight, unified, learnable detector that models a frozen LVLM's heterogeneous computational trace from a single forward pass.
  arXiv  Detail & Related papers  (2026-08-11T12:01:59Z)
• See Only When Needed: Context-Aware Attention Intervention for Mitigating Hallucinations in LVLMs [81.71384150441163]
  We propose Context-aware Attention Intervention (CAI) for large vision-language models (LVLMs)<n>CAI enforces a see only when needed principle via two-axis selectivity: where to look and when to intervene.<n> experiments across multiple LVLM backbones and benchmarks show that CAI achieves state-of-the-art hallucination mitigation.
  arXiv  Detail & Related papers  (2026-06-29T06:35:47Z)
• Where to Refine, When to Stop: Rethinking Redundancy via Latent Discrepancy for Efficient Visual Autoregressive Generation [50.139984798361375]
  We propose a training-free framework that removes redundancy via latent discrepancy.<n> LD-Pruning substantially reduces inference while maintaining high generation quality, achieving up to 2.35x speedup on Infinity-8B.
  arXiv  Detail & Related papers  (2026-05-29T19:34:39Z)
• CHASD: Language Increment-Calibrated Contrastive Decoding against Hallucination in LVLMs [16.77211535169488]
  We propose Contrastive Hallucination-Aware Step-wise Decoding (CHASD) for Large Vision-Language Models.<n> CHASD uses an uncertainty-driven confidence gate to activate the contrastive branch only when the maximum probability of the next-token is less than the threshold.<n> Experiments show that CHASD improves hallucination-related metrics over strong training-free baselines with competitive inference efficiency.
  arXiv  Detail & Related papers  (2026-05-22T08:03:12Z)
• STEAR: Layer-Aware Spatiotemporal Evidence Intervention for Hallucination Mitigation in Video Large Language Models [14.848157882117613]
  Large Video Language Models (Video-LLMs) are prone to hallucinations, often generating visually unsupported or incorrect temporal relations.<n>We propose STEAR, which identifies high-risk decoding steps and selects token-conditioned visual evidence from grounding-aware middle layers.<n>Experiments demonstrate that STEAR consistently reduces hallucinations while improving temporal consistency, faithfulness, and robustness.
  arXiv  Detail & Related papers  (2026-04-03T13:52:57Z)
• Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigation [51.743225614196774]
  Multimodal large language models (MLLMs) have achieved remarkable progress in vision-language reasoning.<n>They remain vulnerable to hallucination, where generated content deviates from visual evidence.<n>Recent vision enhancement methods attempt to address this issue by reinforcing visual tokens during decoding.<n>We propose Adaptive Visual Reinforcement (AIR), a training-free framework for MLLMs.
  arXiv  Detail & Related papers  (2026-02-27T14:18:51Z)
• CrystaL: Spontaneous Emergence of Visual Latents in MLLMs [55.34169914483764]
  We propose CrystaL (Crystallized Latent Reasoning), a single-stage framework with two paths to process intact and corrupted images.<n>By explicitly aligning the attention patterns and prediction distributions across the two paths, CrystaL crystallizes latent representations into task-relevant visual semantics.<n>Experiments on perception-intensive benchmarks demonstrate that CrystaL consistently outperforms state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-02-24T15:01:30Z)
• SDCD: Structure-Disrupted Contrastive Decoding for Mitigating Hallucinations in Large Vision-Language Models [4.677212795400693]
  Bag-of-Patches behavior of Visions under weak structural supervision acts as a contributing factor of object hallucinations.<n>We introduce a training-free algorithm called Structure-Disrupted Contrastive Decoding (SDCD)<n>By penalizing tokens that maintain high confidence under this structure-less view, SDCD effectively suppresses the texture-driven bias.
  arXiv  Detail & Related papers  (2026-01-07T01:27:58Z)
• Revealing Perception and Generation Dynamics in LVLMs: Mitigating Hallucinations via Validated Dominance Correction [59.801614364841775]
  Large Vision-Language Models (LVLMs) have shown remarkable capabilities, yet hallucinations remain a persistent challenge.<n>This work presents a systematic analysis of the internal evolution of visual perception and token generation in LVLMs.<n>We devise the VDC (d Dominance Correction) strategy, which detects unsupported tokens and replaces them with validated ones to improve output reliability.
  arXiv  Detail & Related papers  (2025-12-21T17:05:42Z)
• Adaptive Residual-Update Steering for Low-Overhead Hallucination Mitigation in Large Vision Language Models [13.32858759983739]
  Large Vision-Language Models (LVLMs) often suffer from object hallucination, generating text inconsistent with visual inputs.<n>Existing inference-time interventions to mitigate this issue present a challenging trade-off.<n>We present Residual-Update Directed DEcoding Regulation (RUDDER), a framework that steers LVLMs towards visually-grounded generation.
  arXiv  Detail & Related papers  (2025-11-13T13:29:38Z)
• Pyramid Token Pruning for High-Resolution Large Vision-Language Models via Region, Token, and Instruction-Guided Importance [60.028070589466445]
  Pyramid Token Pruning (PTP) is a training-free strategy that hierarchically integrates bottom-up visual saliency at both region and token levels with top-down instruction-guided relevance.<n>We show that PTP substantially reduces computational cost, memory usage, and inference latency, with negligible performance degradation.
  arXiv  Detail & Related papers  (2025-09-19T07:28:17Z)
• Mitigating Hallucination of Large Vision-Language Models via Dynamic Logits Calibration [8.192590936983347]
  Large Vision-Language Models (LVLMs) have demonstrated significant advancements in multimodal understanding.<n>They are frequently hampered by hallucination-the generation of text that contradicts visual input.<n>Existing training-free decoding strategies exhibit critical limitations.<n>This paper introduces Dynamic Logits (DLC), a novel training-free decoding framework designed to align text generation with visual evidence at inference time.
  arXiv  Detail & Related papers  (2025-06-26T17:35:40Z)
• Multimodal LLM-Guided Semantic Correction in Text-to-Image Diffusion [52.315729095824906]
  MLLM Semantic-Corrected Ping-Pong-Ahead Diffusion (PPAD) is a novel framework that introduces a Multimodal Large Language Model (MLLM) as a semantic observer during inference.<n>It performs real-time analysis on intermediate generations, identifies latent semantic inconsistencies, and translates feedback into controllable signals that actively guide the remaining denoising steps.<n>Extensive experiments demonstrate PPAD's significant improvements.
  arXiv  Detail & Related papers  (2025-05-26T14:42:35Z)
• Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMs [51.93737995405164]
  Large Vision-Language Models (LVLMs) are susceptible to hallucinations.<n>We introduce a novel Conditional Pointwise Mutual Information (C-PMI) calibrated decoding strategy.<n>We show that the proposed method significantly reduces hallucinations in LVLMs while preserving decoding efficiency.
  arXiv  Detail & Related papers  (2025-05-26T08:36:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.