Reliability Challenges in Diffusion Vision-Language Models

• URL: http://arxiv.org/abs/2609.01318v1
• Date: Tue, 01 Sep 2026 14:38:50 GMT
• Title: Reliability Challenges in Diffusion Vision-Language Models
• Abstract summary: Diffusion-based Large Vision-Language Models (dLVLMs) have emerged as a compelling alternative to autoregressive (AR) LVLMs.<n>We present the first systematic reliability evaluation of hallucination and bias in dLVLMs.
• Score: 2.6281187565150526
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Diffusion-based Large Vision-Language Models (dLVLMs) have recently emerged as a compelling alternative to autoregressive (AR) LVLMs, offering advantages in parallel decoding, bidirectional context, and controllable generation. Despite rapid progress, their reliability properties remain largely uncharacterized. We present the first systematic reliability evaluation of hallucination and bias in dLVLMs, benchmarking six diffusion models against competitive AR baselines across four dimensions. Our key findings are: (1) dLVLMs reverse the yes-bias of AR models in binary visual queries; (2) they achieve competitive hallucination rates yet exhibit degraded linguistic quality; (3) they collapse to near-zero accuracy on underrepresented racial groups with opposite-polarity gender bias; and (4) they exhibit accuracy collapse in multiple-choice settings when the correct option is shorter than its distractors, associated with a length prior that emerges at the first denoising step. Tokens committed at late denoising steps with low confidence further correlate with hallucinated content, pointing to a mechanistic signal unique to diffusion generation. These patterns vary across model families, suggesting reliability is shaped by the generative paradigm together with training data.

Related papers

• Escaping Confidence Trap: Evolutionary Decoding for Mathematical Reasoning in Diffusion LLMs [69.52853771994451]
  We analyze the decoding trajectories of LLaDA 2.0 and identify a recurring diffusion confidence trap.<n>We propose Evolutionary Decoding, a training-free test-time scaling framework that views diffusion decoding as an evolutionary process.
  arXiv  Detail & Related papers  (2026-08-01T11:48:25Z)
• MM-Snowball: Evaluating and Mitigating Hallucination Snowballing in Multimodal Multi-Turn Dialogue [98.58433916474472]
  We introduce MM-Snowball, the first benchmark for diagnosis of hallucination snowballing within dialogues.<n>We propose Conflict-Aware Visual Rectification (CAVR) to counteract this degradation.
  arXiv  Detail & Related papers  (2026-05-30T08:53:34Z)
• Self-Prophetic Decoding to Unlock Visual Search in LVLMs [108.77389957341586]
  Large Vision-Language Models (LVLMs) are rapidly evolving toward true multimodal reasoning.<n>LVLM visual search faces two key challenges: incompatibility among intrinsic capabilities after post-training, and interference in long multi-step reasoning contexts.<n>We introduce SeProD, a self-prophetic decoding framework that leverages intrinsic single-step capabilities to enable coherent multi-step reasoning.
  arXiv  Detail & Related papers  (2026-05-27T17:01:39Z)
• Uncertainty Quantification for Large Language Diffusion Models [35.87984313223733]
  Large Language Diffusion Models (LLDMs) are emerging as an alternative to autoregressive models.<n>Existing uncertainty quantification (UQ) methods assume autoregressive factorization or use expensive repeated sampling.<n>We present the first systematic study of UQ for LLDMs and propose lightweight, zero-shot uncertainty signals.
  arXiv  Detail & Related papers  (2026-05-14T08:39:56Z)
• Lost in Diffusion: Uncovering Hallucination Patterns and Failure Modes in Diffusion Large Language Models [10.020835642521897]
  We present the first controlled comparative study to evaluate hallucination patterns in dLLMs.<n>Our results demonstrate that current dLLMs exhibit a higher propensity for hallucination than AR counterparts controlled for architecture, scale, and pre-training weights.<n>Our findings underscore that although dLLMs have narrowed the performance gap on general tasks, their distinct hallucination mechanisms pose a critical challenge to model reliability.
  arXiv  Detail & Related papers  (2026-04-12T09:59:41Z)
• Hallucination Begins Where Saliency Drops [18.189047289404325]
  hallucinations frequently arise when preceding output tokens exhibit low saliency toward the prediction of the next token.<n>We introduce LVLMs-Saliency, a gradient-aware diagnostic framework that quantifies the visual grounding strength of each output token.<n>Our method significantly reduces hallucination rates while preserving fluency and task performance, offering a robust and interpretable solution.
  arXiv  Detail & Related papers  (2026-01-28T05:50:52Z)
• Ground What You See: Hallucination-Resistant MLLMs via Caption Feedback, Diversity-Aware Sampling, and Conflict Regularization [38.469173375694076]
  This paper systematically analyzes the root causes of hallucinations in Multimodal Large Language Models (MLLMs)<n>It identifies three critical factors: (1) an over-reliance on chained visual reasoning, where inaccurate initial descriptions anchor subsequent inferences to incorrect premises; (2) insufficient exploration diversity during policy optimization, leading the model to generate overly confident but erroneous outputs; and (3) destructive conflicts between training samples, where NTK similarity causes false associations and unstable parameter updates.<n> Experimental results demonstrate that our proposed method significantly reduces hallucination rates and effectively enhances the inference accuracy of MLLMs.
  arXiv  Detail & Related papers  (2026-01-09T07:59:18Z)
• Revealing Perception and Generation Dynamics in LVLMs: Mitigating Hallucinations via Validated Dominance Correction [59.801614364841775]
  Large Vision-Language Models (LVLMs) have shown remarkable capabilities, yet hallucinations remain a persistent challenge.<n>This work presents a systematic analysis of the internal evolution of visual perception and token generation in LVLMs.<n>We devise the VDC (d Dominance Correction) strategy, which detects unsupported tokens and replaces them with validated ones to improve output reliability.
  arXiv  Detail & Related papers  (2025-12-21T17:05:42Z)
• HADSF: Aspect Aware Semantic Control for Explainable Recommendation [4.75127493865044]
  Recent advances in large language models (LLMs) promise more effective information extraction for recommender systems.<n>We propose a two-stage approach that induces a compact, corpus-level aspect vocabulary via adaptive selection and then performs vocabulary-guided, explicitly constrained extraction of structured aspect-opinion triples.<n> Experiments on approximately 3 million reviews spanning 1.5B-70B parameters show that, when integrated into standard rating predictors, HADSF yields consistent reductions in prediction error.
  arXiv  Detail & Related papers  (2025-10-30T20:49:33Z)
• From Denoising to Refining: A Corrective Framework for Vision-Language Diffusion Model [72.73512218682187]
  We introduce ReDiff, a refining-enhanced diffusion framework that teaches the model to identify and correct its own errors.<n>Our approach features a two-stage training process: first, we instill a foundational revision capability by training the model to revise synthetic errors; second, we implement a novel online self-correction loop.<n>This mistake-driven learning endows the model with the crucial ability to revisit and refine its already generated output, effectively breaking the error cascade.
  arXiv  Detail & Related papers  (2025-10-22T06:58:55Z)
• Multimodal LLM-Guided Semantic Correction in Text-to-Image Diffusion [52.315729095824906]
  MLLM Semantic-Corrected Ping-Pong-Ahead Diffusion (PPAD) is a novel framework that introduces a Multimodal Large Language Model (MLLM) as a semantic observer during inference.<n>It performs real-time analysis on intermediate generations, identifies latent semantic inconsistencies, and translates feedback into controllable signals that actively guide the remaining denoising steps.<n>Extensive experiments demonstrate PPAD's significant improvements.
  arXiv  Detail & Related papers  (2025-05-26T14:42:35Z)
• Video SimpleQA: Towards Factuality Evaluation in Large Video Language Models [77.96693360763925]
  Video SimpleQA is the first comprehensive benchmark tailored for factuality evaluation in video contexts.<n>Our work differs from existing video benchmarks through the following key features: Knowledge required: demanding integration of external knowledge beyond the video's explicit narrative.<n>Short-form definitive answer: Answers are crafted as unambiguous and definitively correct in a short format with minimal scoring variance.
  arXiv  Detail & Related papers  (2025-03-24T17:46:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.