Stability-Weighted Decoding for Diffusion Language Models

• URL: http://arxiv.org/abs/2604.17068v1
• Date: Sat, 18 Apr 2026 17:04:10 GMT
• Title: Stability-Weighted Decoding for Diffusion Language Models
• Abstract summary: Existing decoding strategies rely on static confidence metrics computed at a single denoising step, ignoring temporal history and often leading to premature unmasking of unstable tokens.<n>We propose Stability-Weighted Decoding (SWD), a training-free, plug-and-play strategy that incorporates temporal stability into token scoring and acts as a universal modulator for arbitrary score-based decoding policies.
• Score: 8.670026899042483
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Diffusion large language models (dLLMs) enable parallel text generation by iteratively denoising a fully masked sequence, unmasking a subset of masked tokens at each step. Existing decoding strategies rely on static confidence metrics computed at a single denoising step, ignoring temporal history and often leading to premature unmasking of unstable tokens. In this work, we theoretically establish that a token's temporal instability, quantified by the KL divergence between consecutive prediction distributions, provides a strict lower bound on its mutual information with the remaining masked context, indicating that temporally unstable tokens are inherently unsafe to unmask. Based on this insight, we propose Stability-Weighted Decoding (SWD), a training-free, plug-and-play strategy that incorporates temporal stability into token scoring and acts as a universal modulator for arbitrary score-based decoding policies. Experiments on code generation and mathematical reasoning benchmarks demonstrate that SWD consistently improves generation accuracy across representative scoring metrics and selection policies, and exhibits exceptional robustness, maintaining a significant performance lead over standard baselines across varying acceleration ratios.

Related papers

• Supportive Token Revealing for Fast Diffusion Language Model Decoding [10.022716564968695]
  AXON is a training-free module that can be added on top of existing parallel decoding strategies for diffusion language models.<n>We show that AXON improves the quality-latency trade-off of existing parallel decoders, often reducing the number of function evaluations while maintaining or improving accuracy.
  arXiv  Detail & Related papers  (2026-06-02T21:35:54Z)
• SimSD: Simple Speculative Decoding in Diffusion Language Models [61.33773959352141]
  Diffusion large language models (dLLMs) offer faster inference through parallel or blockwise decoding.<n>Their masked language modeling formulation remains incompatible with standard token-level speculative decoding.<n>We propose a speculative decoding algorithm for dLLMs, named SimSD, that equips dLLMs with temporally valid token-level contexts.<n>Our method achieves up to 7.46x higher decoding throughput while maintaining and even improving average generation quality.
  arXiv  Detail & Related papers  (2026-06-01T17:46:46Z)
• STDec: Spatio-Temporal Stability Guided Decoding for dLLMs [49.55447757907809]
  Large Diffusion Language Models (dLLMs) have achieved rapid progress, viewed as a promising alternative to the autoregressive paradigm.<n>Most dLLM decoders still adopt a global confidence threshold, and do not model local context from neighboring decoded states or temporal consistency of predicted token IDs.<n>We observe strong stability neartemporal stability in dLLM decoding, and our approach includes spatial-aware decoding and temporal-aware decoding.
  arXiv  Detail & Related papers  (2026-04-07T18:13:31Z)
• Confidence-Based Decoding is Provably Efficient for Diffusion Language Models [8.527157686215117]
  Diffusion language models (DLMs) have emerged as a promising alternative to autoregressive (AR) models for language modeling.<n>We develop the first theoretical analysis framework for confidence-based decoding in DLMs.
  arXiv  Detail & Related papers  (2026-03-23T17:43:21Z)
• DyLLM: Efficient Diffusion LLM Inference via Saliency-based Token Selection and Partial Attention [2.7422645382944935]
  We present DyLLM, a training-free inference framework that accelerates decoding by selectively computing only salient tokens.<n>Across diverse reasoning and code-generation benchmarks, DyLLM achieves up to 9.6x higher throughput.
  arXiv  Detail & Related papers  (2026-03-09T07:02:01Z)
• CORE: Context-Robust Remasking for Diffusion Language Models [51.59514489363897]
  We propose Context-Robust Remasking (CORE), a training-free framework for inference-time revision.<n>Rather than trusting static token probabilities, CORE identifies context-brittle tokens by probing their sensitivity to targeted masked-context perturbations.<n>On LLaDA-8B-Base, CORE delivers consistent improvements across reasoning and code benchmarks, outperforming compute-matched baselines and improving MBPP by up to 9.2 percentage points.
  arXiv  Detail & Related papers  (2026-02-04T00:12:30Z)
• Accelerate Speculative Decoding with Sparse Computation in Verification [49.74839681322316]
  Speculative decoding accelerates autoregressive language model inference by verifying multiple draft tokens in parallel.<n>Existing sparsification methods are designed primarily for standard token-by-token autoregressive decoding.<n>We propose a sparse verification framework that jointly sparsifies attention, FFN, and MoE components during the verification stage to reduce the dominant computation cost.
  arXiv  Detail & Related papers  (2025-12-26T07:53:41Z)
• STDD:Spatio-Temporal Dynamics-Driven Token Refinement in Diffusion Language Models [12.172699141988728]
  diffusion language models (DLMs) generate text by iteratively denoising all token positions in parallel.<n>We propose a novel remasking approach that dynamically detects Temporal Variance and Spatial Deviance of each token.<n>Our approach significantly improves the operational efficiency of DLMs across mainstream datasets, achieving speedups of up to 8.9 times.
  arXiv  Detail & Related papers  (2025-12-07T12:53:48Z)
• Continuously Augmented Discrete Diffusion model for Categorical Generative Modeling [87.34677262370924]
  Standard discrete diffusion models treat all unobserved states identically by mapping them to an absorbing [MASK] token.<n>This creates an 'information void' where semantic information that could be inferred from unmasked tokens is lost between denoising steps.<n>We introduce Continuously Augmented Discrete Diffusion, a framework that augments the discrete state space with a paired diffusion in a continuous latent space.
  arXiv  Detail & Related papers  (2025-10-01T18:00:56Z)
• StableToken: A Noise-Robust Semantic Speech Tokenizer for Resilient SpeechLLMs [54.229363096087866]
  Speech tokenizers are not robust to meaning-irrelevant acoustic perturbations.<n>This instability stems from two flaws: a brittle single-path quantization architecture and a distant training signal.<n>We introduce StableToken, a tokenizer that achieves stability through a consensus-driven mechanism.
  arXiv  Detail & Related papers  (2025-09-26T11:32:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.