Discrete Stochastic Localization for Non-autoregressive Generation

• URL: http://arxiv.org/abs/2605.12836v2
• Date: Wed, 20 May 2026 20:19:39 GMT
• Title: Discrete Stochastic Localization for Non-autoregressive Generation
• Abstract summary: We argue that the bottleneck is not continuity itself, but a representation in which denoising depends on timestep-indexed noise regimes.<n>We introduce emphDiscrete, a continuous-state framework with unit-sphere token embeddings.<n>One trained network then supports an entire family of per-token SNR paths, with endpoint masked-diffusion paths as a special case.
• Score: 18.211690435751446
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Continuous diffusion is a natural framework for non-autoregressive generation but has generally lagged behind masked discrete diffusion models (MDMs) on discrete sequence generation. We argue that the bottleneck is not continuity itself, but a representation in which denoising depends on timestep-indexed noise regimes. We introduce \emph{Discrete Stochastic Localization} (DSL), a continuous-state framework with unit-sphere token embeddings whose Bayes-optimal denoiser is invariant to the nominal signal-to-noise ratio (SNR) under the localization channel. One trained network then supports an entire family of per-token SNR paths, with endpoint masked-diffusion paths as a special case. Fine-tuning a pretrained MDLM checkpoint with DSL substantially improves distributional faithfulness (MAUVE) on OpenWebText across all step budgets from $T{=}128$ to $T{=}1024$, and the same checkpoint supports random-order autoregressive sampling, as well as a hybrid continuous-then-discrete sampler using as few as T=48 total steps -- without distillation or retraining.

Related papers

• TEMPER: Tensorized Efficient Manifold-constrained Parameterization for Expressive Residual Routing [7.909872430050534]
  Residual connections rely on a static residual pathway, and are essential for training deep neural networks.<n>We propose underlinetextbfTensorized underlinetextbfEfficient underlinetextbfManifold-constrained underlinetextbfParameterization for underlinetextbfExpressive Residual underlinetextbfRouting.
  arXiv  Detail & Related papers  (2026-08-08T01:45:27Z)
• DSL-LLaDA: Scaling Continuous Denoising to 8B Masked Diffusion LMs [29.561307941602482]
  We show that a pretrained masked DLM can be lightly adapted to support continuous embedding-space denoising.<n>The adapted model supports continuous inference that evolves all positions jointly in embedding space.<n> DSL-LLaDA-SDE achieves the best ROUGE-1 on all four benchmarks.
  arXiv  Detail & Related papers  (2026-05-31T05:27:01Z)
• Neural Continuous-Time Markov Chain: Discrete Diffusion via Decoupled Jump Timing and Direction [18.20305761007799]
  We propose textbfNeural CTMC, which parameterizes the reverse process through an emphexit rate (when to jump) and a emphjump distribution (where to jump) using two dedicated network heads.<n>We show that the evidence lower bound (ELBO) differs from a path-space KL divergence between the true and learned reverse processes by a $$-independent constant.<n>We also show that the tractable conditional surrogate preserves the gradients and minimizers of the corresponding marginal reverse-process objective under standard regularity assumptions.
  arXiv  Detail & Related papers  (2026-04-17T04:40:19Z)
• CRoCoDiL: Continuous and Robust Conditioned Diffusion for Language [21.764828393078886]
  Masked Diffusion Models (MDMs) provide an efficient non-causal alternative to autoregressive generation.<n>We address these limitations by shifting the diffusion process into a continuous sentence-level semantic space.
  arXiv  Detail & Related papers  (2026-03-02T12:50:55Z)
• Rejection Mixing: Fast Semantic Propagation of Mask Tokens for Efficient DLLM Inference [58.189320101488725]
  DLLMs promise fast non-autoregressive inference but suffer a severe quality-speed trade-off in parallel decoding.<n>We address this by integrating continuous representations into the discrete decoding process, as they preserve rich inter-position dependency.<n>We propose ReMix, a framework that introduces a novel Continuous Mixing State as an intermediate between the initial masked state and the final decoded token state.
  arXiv  Detail & Related papers  (2026-02-26T11:08:11Z)
• Discrete Stochastic Localization for Non-autoregressive Generation [17.56505846228918]
  We show that emphtraining alone can substantially improve the step-efficiency of MDLM/ReMDM sampling.<n>On OpenWebText, textsc fine-tuning yields large MAUVE gains at low step budgets, surpassing the MDLM+ReMDM.<n>Analyses show improved self-correction and uncertainty calibration, making remasking markedly more compute-efficient.
  arXiv  Detail & Related papers  (2026-02-18T04:05:40Z)
• Efficient Sampling with Discrete Diffusion Models: Sharp and Adaptive Guarantees [9.180350432640912]
  We study the sampling efficiency of score-based discrete diffusion models under a continuous-time Markov chain (CTMC) formulation.<n>For uniform discrete diffusion, we show that the $$-leaping algorithm achieves an complexity of order $tilde O(d/varepsilon)$.<n>For masking discrete diffusion, we introduce a modified $$-leaping sampler whose convergence rate is governed by an intrinsic information-theoretic quantity.
  arXiv  Detail & Related papers  (2026-02-16T18:48:17Z)
• Self-Rewarding Sequential Monte Carlo for Masked Diffusion Language Models [58.946955321428845]
  This work presents self-rewarding sequential Monte Carlo (SMC)<n>Our algorithm stems from the observation that most existing MDLMs rely on a confidence-based sampling strategy.<n>We introduce the trajectory-level confidence as a self-rewarding signal for assigning particle importance weights.
  arXiv  Detail & Related papers  (2026-02-02T09:21:45Z)
• Variational Bayesian Flow Network for Graph Generation [54.94088904387278]
  We propose Variational Bayesian Flow Network (VBFN) for graph generation.<n>VBFN performs variational lifting to a tractable joint Gaussian variational belief family governed by structured precisions.<n>On synthetic and molecular graph datasets, VBFN improves fidelity and diversity, and surpasses baseline methods.
  arXiv  Detail & Related papers  (2026-01-30T03:59:38Z)
• Stratified Hazard Sampling: Minimal-Variance Event Scheduling for CTMC/DTMC Discrete Diffusion and Flow Models [0.0]
  Stratified Hazard Sampling Sampling (SHS) models per-token edits as events driven by cumulative hazard (CTMC) or cumulative jump mass (DTMC)<n>SHS models per-token edits as events driven by cumulative hazard (CTMC) or cumulative jump mass (DTMC) and places events by stratifying this cumulative quantity.<n>We also introduce a phase-allocation variant for blacklist-style lexical constraints that prioritizes early edits at high-risk positions to mitigate late-masking artifacts.
  arXiv  Detail & Related papers  (2026-01-06T08:19:02Z)
• Continuously Augmented Discrete Diffusion model for Categorical Generative Modeling [87.34677262370924]
  Standard discrete diffusion models treat all unobserved states identically by mapping them to an absorbing [MASK] token.<n>This creates an 'information void' where semantic information that could be inferred from unmasked tokens is lost between denoising steps.<n>We introduce Continuously Augmented Discrete Diffusion, a framework that augments the discrete state space with a paired diffusion in a continuous latent space.
  arXiv  Detail & Related papers  (2025-10-01T18:00:56Z)
• Inference-Time Scaling of Diffusion Language Models with Particle Gibbs Sampling [70.8832906871441]
  We study how to steer generation toward desired rewards without retraining the models.<n>Prior methods typically resample or filter within a single denoising trajectory, optimizing rewards step-by-step without trajectory-level refinement.<n>We introduce particle Gibbs sampling for diffusion language models (PG-DLM), a novel inference-time algorithm enabling trajectory-level refinement while preserving generation perplexity.
  arXiv  Detail & Related papers  (2025-07-11T08:00:47Z)
• ReDDiT: Rehashing Noise for Discrete Visual Generation [53.813067778912]
  We propose a rehashing noise approach for discrete diffusion transformer (termed ReDDiT)<n>We aim to extend absorbing states and improve expressive capacity of discrete diffusion models.<n>Experiments show that ReDDiT significantly outperforms the baseline model.
  arXiv  Detail & Related papers  (2025-05-26T08:17:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.