Consistent Diffusion Language Models

• URL: http://arxiv.org/abs/2605.00161v1
• Date: Thu, 30 Apr 2026 19:31:02 GMT
• Title: Consistent Diffusion Language Models
• Abstract summary: Diffusion language models (DLMs) promise sub-time, parallel generation, yet practical gains remain elusive.<n>In continuous domains, consistency training along the probability-flow ODE is a popular recipe to accelerate diffusion.<n>We introduce Multi-Path Discrete Consistency (DCMP), a new principle that trains a denoiser to be path-in expectation across these bridges.<n>A single CDLM objective unifies masked diffusion, continuous consistency models, and progressive/discrete distillation as analytic limits or empirical approximations of one common view.
• Score: 22.253153649144476
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Diffusion language models (DLMs) are an attractive alternative to autoregressive models because they promise sublinear-time, parallel generation, yet practical gains remain elusive as high-quality samples still demand hundreds of refinement steps. In continuous domains, consistency training along the probability-flow ODE is a popular recipe to accelerate diffusion. For discrete diffusion, no analogous sample-space ODE exists, making direct adaptation ill-defined. We argue that the natural discrete substitute is not a deterministic trajectory but its stochastic counterpart: the exact posterior bridge, available in closed form for broad corruption families including masked and uniform diffusion. Building on this observation, we introduce Multi-Path Discrete Consistency (MPDC), a new principle that trains a denoiser to be path-invariant in expectation across these stochastic bridges, and instantiate it as the Consistent Diffusion Language Model (CDLM), a single-stage, teacher-free training framework. A single CDLM objective unifies masked diffusion, continuous consistency models, and progressive/discrete distillation as analytic limits or empirical approximations of one common view. Empirically, CDLM establishes a new state of the art on both conditional and unconditional text-generation, consistently outperforming strong base discrete diffusion models and often even multi-stage distilled baselines across sampling budgets, with the largest gains in the few-step regime. Together, these results position CDLM as a principled and scalable foundation for the next generation of fast, high-fidelity discrete generative modeling.

Related papers

• Tensor-Train Joint Modeling for Few-Step Discrete Diffusion [25.442970567297326]
  We present a framework for explicit joint distribution modeling in discrete diffusion via tensor decomposition.<n>Our framework integrates into pretrained MDMs through lightweight fine-tuning, yielding substantial improvements in few-step generation at a fraction of the cost of training from scratch.
  arXiv  Detail & Related papers  (2026-07-04T09:32:12Z)
• Interpolating Discrete Diffusion Models with Controllable Resampling [49.771004594828405]
  Masked diffusion models suffer from irreversible errors due to early unmasking.<n>Uniform diffusion models, despite enabling self-correction, often yield low-quality samples.<n>We introduce Interpolating Discrete Diffusion Model, that improves diffusion by reducing dependence on intermediate latent states.
  arXiv  Detail & Related papers  (2026-04-19T07:55:57Z)
• Generalized Discrete Diffusion from Snapshots [18.212406369597662]
  GDDS is a unified framework for discrete diffusion modeling.<n>It supports arbitrary noising processes over large discrete state spaces.<n>For the reverse process, we derive a simple evidence lower bound (ELBO) based on snapshot latents.
  arXiv  Detail & Related papers  (2026-03-22T17:58:01Z)
• Bridging the Discrete-Continuous Gap: Unified Multimodal Generation via Coupled Manifold Discrete Absorbing Diffusion [60.186310080523135]
  Bifurcation of generative modeling into autoregressive approaches for discrete data (text) and diffusion approaches for continuous data (images) hinders development of truly unified multimodal systems.<n>We propose textbfCoM-DAD, a novel probabilistic framework that reformulates multimodal generation as a hierarchical dual-process.<n>Our method demonstrates superior stability over standard masked modeling, establishing a new paradigm for scalable, unified text-image generation.
  arXiv  Detail & Related papers  (2026-01-07T16:21:19Z)
• Foundations of Diffusion Models in General State Spaces: A Self-Contained Introduction [54.95522167029998]
  This article is a self-contained primer on diffusion over general state spaces.<n>We develop the discrete-time view (forward noising via Markov kernels and learned reverse dynamics) alongside its continuous-time limits.<n>A common variational treatment yields the ELBO that underpins standard training losses.
  arXiv  Detail & Related papers  (2025-12-04T18:55:36Z)
• Coevolutionary Continuous Discrete Diffusion: Make Your Diffusion Language Model a Latent Reasoner [66.86440230599656]
  We argue that diffusion language models do not necessarily need to be in the discrete space.<n>In particular, we prove that continuous diffusion models have stronger expressivity than discrete diffusions and looped transformers.<n>We propose Coevolutionary Continuous Diffusion (CCDD), which defines a joint multimodal diffusion process on the union of a continuous representation space and a discrete token space.
  arXiv  Detail & Related papers  (2025-10-03T17:44:41Z)
• Forward-only Diffusion Probabilistic Models [23.247850964456177]
  This work presents a forward-only diffusion (FoD) approach for generative modelling.<n>FoD directly learns data generation through a single forward diffusion process, yielding a simple yet efficient generative framework.<n>Our code is available at https://github.com/Algolzw/FoD.
  arXiv  Detail & Related papers  (2025-05-22T14:47:07Z)
• Constrained Discrete Diffusion [61.81569616239755]
  This paper introduces Constrained Discrete Diffusion (CDD), a novel integration of differentiable constraint optimization within the diffusion process.<n>CDD directly imposes constraints into the discrete diffusion sampling process, resulting in a training-free and effective approach.
  arXiv  Detail & Related papers  (2025-03-12T19:48:12Z)
• Continuous Diffusion Model for Language Modeling [64.7425225935854]
  Existing continuous diffusion models for discrete data underperform compared to discrete methods.<n>We propose a continuous diffusion model for language modeling that incorporates the geometry of the underlying categorical distribution.<n>Our method outperforms existing discrete diffusion models and approaches the performance of autoregressive models.
  arXiv  Detail & Related papers  (2025-02-17T08:54:29Z)
• Improving Vector-Quantized Image Modeling with Latent Consistency-Matching Diffusion [55.185588994883226]
  We introduce VQ-LCMD, a continuous-space latent diffusion framework within the embedding space that stabilizes training.<n>VQ-LCMD uses a novel training objective combining the joint embedding-diffusion variational lower bound with a consistency-matching (CM) loss.<n>Experiments show that the proposed VQ-LCMD yields superior results on FFHQ, LSUN Churches, and LSUN Bedrooms compared to discrete-state latent diffusion models.
  arXiv  Detail & Related papers  (2024-10-18T09:12:33Z)
• Convergence Analysis of Discrete Diffusion Model: Exact Implementation through Uniformization [17.535229185525353]
  We introduce an algorithm leveraging the uniformization of continuous Markov chains, implementing transitions on random time points. Our results align with state-of-the-art achievements for diffusion models in $mathbbRd$ and further underscore the advantages of discrete diffusion models in comparison to the $mathbbRd$ setting.
  arXiv  Detail & Related papers  (2024-02-12T22:26:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.