Continuous Diffusion Scales Competitively with Discrete Diffusion for Language

• URL: http://arxiv.org/abs/2605.18530v1
• Date: Mon, 18 May 2026 15:15:24 GMT
• Title: Continuous Diffusion Scales Competitively with Discrete Diffusion for Language
• Abstract summary: We construct RePlaid as a likelihood-based continuous diffusion language model (DLM)<n>We show that RePlaid exhibits a compute gap of only $20times$ compared to autoregressive models.<n>We also offer theoretical insights to understand the advantage of likelihood-based training.
• Score: 48.290726603587444
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While diffusion has drawn considerable recent attention from the language modeling community, continuous diffusion has appeared less scalable than discrete approaches. To challenge this belief we revisit Plaid, a likelihood-based continuous diffusion language model (DLM), and construct RePlaid by aligning the architecture of Plaid with modern discrete DLMs. In this unified setting, we establish the first scaling law for continuous DLMs that rivals discrete DLMs: RePlaid exhibits a compute gap of only $20\times$ compared to autoregressive models, outperforms Duo while using fewer parameters, and outperforms MDLM in the over-trained regime. We benchmark RePlaid against recent continuous DLMs: on OpenWebText, RePlaid achieves a new state-of-the-art PPL bound of $22.1$ among continuous DLMs and superior generation quality. These results suggest that continuous diffusion, when trained via likelihood, is a highly competitive and scalable alternative to discrete DLMs. Moreover, we offer theoretical insights to understand the advantage of likelihood-based training. We show that optimizing the noise schedule to minimize the ELBO's variance naturally yields linear cross-entropy (information loss) over time. This evenly distributes denoising difficulty without any case-specific time reparameterization. In addition, we find that optimizing embeddings via likelihood creates structured geometries and drives the most significant likelihood gain.

Related papers

• Tensor-Train Joint Modeling for Few-Step Discrete Diffusion [25.442970567297326]
  We present a framework for explicit joint distribution modeling in discrete diffusion via tensor decomposition.<n>Our framework integrates into pretrained MDMs through lightweight fine-tuning, yielding substantial improvements in few-step generation at a fraction of the cost of training from scratch.
  arXiv  Detail & Related papers  (2026-07-04T09:32:12Z)
• Semantic DLM+: Improving Diffusion Language Models through Bias-variance Trade-off in Transition Kernel Design [59.05127237532803]
  Diffusion Language Models (DLMs) have demonstrated strong scaling capacity as alternatives to autoregressive language models.<n>We study this sensitivity through a principled analysis of generalization error and identify three critical factors.<n>We propose SemDLM+, which adds a global transition and a semantic-frequency penalty during sampling.
  arXiv  Detail & Related papers  (2026-06-13T14:41:55Z)
• ELF: Embedded Language Flows [76.80499600969769]
  We show that continuous language models can be made effective with minimal adaptation to the discrete domain.<n>We propose Embedded Language Flows (ELF), a class of diffusion models in continuous embedding space.<n>ELF substantially outperforms leading discrete and continuous DLMs, achieving better generation quality with fewer sampling steps.
  arXiv  Detail & Related papers  (2026-05-11T17:59:29Z)
• Consistent Diffusion Language Models [22.253153649144476]
  Diffusion language models (DLMs) promise sub-time, parallel generation, yet practical gains remain elusive.<n>In continuous domains, consistency training along the probability-flow ODE is a popular recipe to accelerate diffusion.<n>We introduce Multi-Path Discrete Consistency (DCMP), a new principle that trains a denoiser to be path-in expectation across these bridges.<n>A single CDLM objective unifies masked diffusion, continuous consistency models, and progressive/discrete distillation as analytic limits or empirical approximations of one common view.
  arXiv  Detail & Related papers  (2026-04-30T19:31:02Z)
• One-step Language Modeling via Continuous Denoising [36.18484491074519]
  We show that language models leveraging flow-based continuous denoising can outperform discrete diffusion in both quality and speed.<n>Our work calls into question the widely held hypothesis that discrete diffusion processes are necessary for generative modeling over discrete modalities.
  arXiv  Detail & Related papers  (2026-02-18T19:23:07Z)
• Not All Denoising Steps Are Equal: Model Scheduling for Faster Masked Diffusion Language Models [6.4453302264198165]
  We exploit the flexibility of the diffusion framework and study model scheduling, where a smaller MDLM replaces the full model at a subset of denoising steps.<n>We show that early and late denoising steps are substantially more robust to such replacement than middle steps, enabling up to a 17% reduction in FLOPs.<n>Our results suggest that simple, architecture-agnostic scheduling rules can significantly accelerate MDLM sampling while largely preserving generation quality.
  arXiv  Detail & Related papers  (2026-02-04T13:04:58Z)
• Causal Autoregressive Diffusion Language Model [70.7353007255797]
  CARD reformulates the diffusion process within a strictly causal attention mask, enabling dense, per-token supervision in a single forward pass.<n>Our results demonstrate that CARD achieves ARM-level data efficiency while unlocking the latency benefits of parallel generation.
  arXiv  Detail & Related papers  (2026-01-29T17:38:29Z)
• Bridging the Discrete-Continuous Gap: Unified Multimodal Generation via Coupled Manifold Discrete Absorbing Diffusion [60.186310080523135]
  Bifurcation of generative modeling into autoregressive approaches for discrete data (text) and diffusion approaches for continuous data (images) hinders development of truly unified multimodal systems.<n>We propose textbfCoM-DAD, a novel probabilistic framework that reformulates multimodal generation as a hierarchical dual-process.<n>Our method demonstrates superior stability over standard masked modeling, establishing a new paradigm for scalable, unified text-image generation.
  arXiv  Detail & Related papers  (2026-01-07T16:21:19Z)
• Latent Discrete Diffusion Models [18.979326092796896]
  We study discrete diffusion for language and other categorical data.<n>We propose emphLatent Discrete Diffusion Models (LDDM)<n>We present two instantiations: (i) FUJI-LDDMs, which perform fully joint denoising of tokens and latents, and (ii) SEQ-LDDMs, which sequentially resolve the latent and then the discrete chain conditionally on it.<n>For both variants we derive ELBO-style objectives and discuss design choices to learn informative latents yet amenable to diffusoin modeling.
  arXiv  Detail & Related papers  (2025-10-20T21:26:52Z)
• Improving Vector-Quantized Image Modeling with Latent Consistency-Matching Diffusion [55.185588994883226]
  We introduce VQ-LCMD, a continuous-space latent diffusion framework within the embedding space that stabilizes training.<n>VQ-LCMD uses a novel training objective combining the joint embedding-diffusion variational lower bound with a consistency-matching (CM) loss.<n>Experiments show that the proposed VQ-LCMD yields superior results on FFHQ, LSUN Churches, and LSUN Bedrooms compared to discrete-state latent diffusion models.
  arXiv  Detail & Related papers  (2024-10-18T09:12:33Z)
• Unified Discrete Diffusion for Categorical Data [37.56355078250024]
  We present a series of mathematical simplifications of the variational lower bound that enable more accurate and easy-to-optimize training for discrete diffusion. We derive a simple formulation for backward denoising that enables exact and accelerated sampling, and importantly, an elegant unification of discrete-time and continuous-time discrete diffusion.
  arXiv  Detail & Related papers  (2024-02-06T04:42:36Z)
• A Cheaper and Better Diffusion Language Model with Soft-Masked Noise [62.719656543880596]
  Masked-Diffuse LM is a novel diffusion model for language modeling, inspired by linguistic features in languages. Specifically, we design a linguistic-informed forward process which adds corruptions to the text through strategically soft-masking to better noise the textual data. We demonstrate that our Masked-Diffuse LM can achieve better generation quality than the state-of-the-art diffusion models with better efficiency.
  arXiv  Detail & Related papers  (2023-04-10T17:58:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.