STDD:Spatio-Temporal Dynamics-Driven Token Refinement in Diffusion Language Models

• URL: http://arxiv.org/abs/2601.04205v1
• Date: Sun, 07 Dec 2025 12:53:48 GMT
• Title: STDD:Spatio-Temporal Dynamics-Driven Token Refinement in Diffusion Language Models
• Abstract summary: diffusion language models (DLMs) generate text by iteratively denoising all token positions in parallel.<n>We propose a novel remasking approach that dynamically detects Temporal Variance and Spatial Deviance of each token.<n>Our approach significantly improves the operational efficiency of DLMs across mainstream datasets, achieving speedups of up to 8.9 times.
• Score: 12.172699141988728
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Unlike autoregressive language models, diffusion language models (DLMs) generate text by iteratively denoising all token positions in parallel. At each timestep, the remasking strategy of a DLM selects low-priority tokens to defer their decoding, thereby improving both efficiency and output quality. However, mainstream remasking strategies rely on a single global confidence threshold, overlooking the temporal and spatial dynamics of individual tokens. Motivated by the redundant iterations and constrained parallelism introduced by fixed-threshold remasking, we propose a novel remasking approach that dynamically detects Temporal Variance and Spatial Deviance of each token, which reflect its convergence status and inter-token correlations. Using these signals, our method adaptively adjusts the confidence threshold for every token at every step. Empirical results show that our approach significantly improves the operational efficiency of DLMs across mainstream datasets, achieving speedups of up to 8.9 times while faithfully preserving generation quality.

Related papers

• AURORA-LM: Autoencoding Unified Representation for Continuous-Latent Diffusion Language Modeling [60.04959047453115]
  AURORA-LM is a continuous-latent language model that separates the construction of a decodable text representation from the modeling of its distribution.<n>AURORA-LM achieves the strongest performance among evaluated continuous and diffusion-based language models on OpenWebText free generation and XSum summarization.
  arXiv  Detail & Related papers  (2026-08-03T17:59:50Z)
• SimSD: Simple Speculative Decoding in Diffusion Language Models [61.33773959352141]
  Diffusion large language models (dLLMs) offer faster inference through parallel or blockwise decoding.<n>Their masked language modeling formulation remains incompatible with standard token-level speculative decoding.<n>We propose a speculative decoding algorithm for dLLMs, named SimSD, that equips dLLMs with temporally valid token-level contexts.<n>Our method achieves up to 7.46x higher decoding throughput while maintaining and even improving average generation quality.
  arXiv  Detail & Related papers  (2026-06-01T17:46:46Z)
• Stability-Weighted Decoding for Diffusion Language Models [8.670026899042483]
  Existing decoding strategies rely on static confidence metrics computed at a single denoising step, ignoring temporal history and often leading to premature unmasking of unstable tokens.<n>We propose Stability-Weighted Decoding (SWD), a training-free, plug-and-play strategy that incorporates temporal stability into token scoring and acts as a universal modulator for arbitrary score-based decoding policies.
  arXiv  Detail & Related papers  (2026-04-18T17:04:10Z)
• Confidence-Based Decoding is Provably Efficient for Diffusion Language Models [8.527157686215117]
  Diffusion language models (DLMs) have emerged as a promising alternative to autoregressive (AR) models for language modeling.<n>We develop the first theoretical analysis framework for confidence-based decoding in DLMs.
  arXiv  Detail & Related papers  (2026-03-23T17:43:21Z)
• DyLLM: Efficient Diffusion LLM Inference via Saliency-based Token Selection and Partial Attention [2.7422645382944935]
  We present DyLLM, a training-free inference framework that accelerates decoding by selectively computing only salient tokens.<n>Across diverse reasoning and code-generation benchmarks, DyLLM achieves up to 9.6x higher throughput.
  arXiv  Detail & Related papers  (2026-03-09T07:02:01Z)
• Just on Time: Token-Level Early Stopping for Diffusion Language Models [0.0]
  Diffusion language models generate text through iterative refinement, a process that is often computationally inefficient.<n>We introduce a training-free, token-level early stopping approach that identifies convergence independently at each position.<n>This yields adaptive per-token freezing without task-specific fine-tuning, substantially reducing the total number of diffusion steps required.
  arXiv  Detail & Related papers  (2026-02-11T18:44:04Z)
• Self-Rewarding Sequential Monte Carlo for Masked Diffusion Language Models [58.946955321428845]
  This work presents self-rewarding sequential Monte Carlo (SMC)<n>Our algorithm stems from the observation that most existing MDLMs rely on a confidence-based sampling strategy.<n>We introduce the trajectory-level confidence as a self-rewarding signal for assigning particle importance weights.
  arXiv  Detail & Related papers  (2026-02-02T09:21:45Z)
• SLM-SS: Speech Language Model for Generative Speech Separation [47.06391017558454]
  We propose SLM-SS, a novel approach that applies speech language models to speech separation.<n>We frame SS as discrete multi-codebook sequence generation, usingcoder models to map quantized speech mixtures to target tokens.<n>Our approach shows significantly better preservation of speech intelligibility, leading to improved linguistic consistency in a variety of downstream tasks.
  arXiv  Detail & Related papers  (2026-01-27T12:22:43Z)
• Learning Unmasking Policies for Diffusion Language Models [33.44995119635116]
  Language Models (dLLMs) now match the downstream performance of their autoregressive counterparts on many tasks.<n>One particularly successful variant is masked discrete diffusion, in which a buffer filled with special mask tokens is progressively replaced with tokens sampled from the model's vocabulary.<n>In this work, we propose to train sampling procedures using reinforcement learning.
  arXiv  Detail & Related papers  (2025-12-09T20:44:33Z)
• Beyond Confidence: Adaptive and Coherent Decoding for Diffusion Language Models [64.92045568376705]
  Coherent Contextual Decoding (CCD) is a novel inference framework built upon two core innovations.<n>CCD employs a trajectory rectification mechanism that leverages historical context to enhance sequence coherence.<n>Instead of rigid allocations based on diffusion steps, we introduce an adaptive sampling strategy that dynamically adjusts the unmasking budget for each step.
  arXiv  Detail & Related papers  (2025-11-26T09:49:48Z)
• Beyond Static Cutoffs: One-Shot Dynamic Thresholding for Diffusion Language Models [3.4202631641965238]
  One-Shot Dynamic Thresholding (OSDT) calibrates thresholds on a single sequence and applies them to subsequent inputs with negligible overhead.<n>Our results suggest broader opportunities to leverage reusable task-level confidence signatures for more general-purpose algorithmic and systems innovations in diffusion decoding.
  arXiv  Detail & Related papers  (2025-11-03T21:30:03Z)
• Growing Visual Generative Capacity for Pre-Trained MLLMs [60.826355079902505]
  Bridge is a pure autoregressive unified MLLM that augments pre-trained visual understanding models with generative ability.<n>We propose a semantic-to-pixel discrete representation that integrates compact semantic tokens with fine-grained pixel tokens.
  arXiv  Detail & Related papers  (2025-10-02T00:40:02Z)
• Reward-Weighted Sampling: Enhancing Non-Autoregressive Characteristics in Masked Diffusion LLMs [44.55861996331439]
  Masked diffusion models (MDMs) offer a promising non-autoregressive alternative for large language modeling.<n>Standard decoding methods for MDMs select tokens independently based on individual token confidences at each diffusion step.<n>We propose Reward-Weighted Sampling (RWS) to provide a principled global signal during the iterative diffusion process.
  arXiv  Detail & Related papers  (2025-08-31T05:48:30Z)
• LayerCake: Token-Aware Contrastive Decoding within Large Language Model Layers [53.43862310647276]
  Large language models (LLMs) excel at natural language understanding and generation but remain vulnerable to factual errors.<n>We introduce a token-aware, layer-localized contrastive decoding method that aligns specific token types with their most influential transformer layers to improve factual generation.<n>Our method requires no additional training or model modification, and experiments demonstrate that our method consistently improves factuality across multiple LLMs and various benchmarks.
  arXiv  Detail & Related papers  (2025-07-06T14:35:43Z)
• FELLE: Autoregressive Speech Synthesis with Token-Wise Coarse-to-Fine Flow Matching [56.30231216917128]
  FELLE is an autoregressive model that integrates language modeling with token-wise flow matching.<n>For each continuous-valued token, FELLE modifies the general prior distribution in flow matching by incorporating information from the previous step.<n>FELLE generates continuous-valued tokens hierarchically, conditioned on the language model's output.
  arXiv  Detail & Related papers  (2025-02-16T13:54:32Z)
• Improve Variational Autoencoder for Text Generationwith Discrete Latent Bottleneck [52.08901549360262]
  Variational autoencoders (VAEs) are essential tools in end-to-end representation learning. VAEs tend to ignore latent variables with a strong auto-regressive decoder. We propose a principled approach to enforce an implicit latent feature matching in a more compact latent space.
  arXiv  Detail & Related papers  (2020-04-22T14:41:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.