Breaking the Factorization Barrier in Diffusion Language Models

• URL: http://arxiv.org/abs/2603.00045v1
• Date: Mon, 09 Feb 2026 08:36:39 GMT
• Title: Breaking the Factorization Barrier in Diffusion Language Models
• Abstract summary: "Factorization barrier" impedes efficient parallel generation of diffusion language models.<n>We propose Coupled Discrete Diffusion to replace the fully-factorized output distribution.<n>We show that CoDD seamlessly enhances diverse diffusion language model architectures with negligible overhead.
• Score: 59.946071582340146
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Diffusion language models theoretically allow for efficient parallel generation but are practically hindered by the "factorization barrier": the assumption that simultaneously predicted tokens are independent. This limitation forces a trade-off: models must either sacrifice speed by resolving dependencies sequentially or suffer from incoherence due to factorization. We argue that this barrier arises not from limited backbone expressivity, but from a structural misspecification: models are restricted to fully factorized outputs because explicitly parameterizing a joint distribution would require the Transformer to output a prohibitively large number of parameters. We propose Coupled Discrete Diffusion (CoDD), a hybrid framework that breaks this barrier by replacing the fully-factorized output distribution with a lightweight, tractable probabilistic inference layer. This formulation yields a distribution family that is significantly more expressive than standard factorized priors, enabling the modeling of complex joint dependencies, yet remains compact enough to avoid the prohibitive parameter explosion associated with full joint modeling. Empirically, CoDD seamlessly enhances diverse diffusion language model architectures with negligible overhead, matching the reasoning performance of computationally intensive Reinforcement Learning baselines at a fraction of the training cost. Furthermore, it prevents performance collapse in few-step generation, enabling high-quality outputs at significantly reduced latencies. Code available at: https://github.com/liuanji/CoDD

Related papers

• Forward-Free Diffusion Language Models [12.961496586646708]
  Diffusion language models generate text through iterative denoising.<n>We propose FReDA, a forward-free diffusion language model that eliminates the need for a hand-designed forward process.<n>FReDA is neighborhood-agnostic, model-complexity-aware, and compatible with flexible refinement parameterizations.
  arXiv  Detail & Related papers  (2026-06-06T22:10:46Z)
• The Expressivity Boundary of Probabilistic Circuits: A Comparison with Large Language Models [49.67598869075125]
  In autoregressive language modeling, Probabilistic Circuits (PCs) lag behind Transformer-based large language models (LLMs)<n>We show that structured-decomposable PCs can match Transformer separation rank on vtree-aligned partitions, but that this capacity is limited to partitions aligned with the fixed routing structure.<n>We further prove that decomposable PCs are strictly more expressive than structured-decomposable ones, though effectively optimizing them remains an open challenge.
  arXiv  Detail & Related papers  (2026-05-13T03:22:10Z)
• Information-Theoretic Limits of Reliability and Scaling in Language Models [6.658166782006281]
  Large language models (LLMs) are evaluated as though perfect reliability is achievable for any task given sufficient scale.<n>We show this assumption is information-theoretically unjustified.<n>Our work formalizes the resource-complexity tradeoffs that govern model performance across domains.
  arXiv  Detail & Related papers  (2026-05-08T06:06:18Z)
• Causal Autoregressive Diffusion Language Model [70.7353007255797]
  CARD reformulates the diffusion process within a strictly causal attention mask, enabling dense, per-token supervision in a single forward pass.<n>Our results demonstrate that CARD achieves ARM-level data efficiency while unlocking the latency benefits of parallel generation.
  arXiv  Detail & Related papers  (2026-01-29T17:38:29Z)
• Repulsor: Accelerating Generative Modeling with a Contrastive Memory Bank [65.00301565190824]
  mname is a plug-and-play training framework that requires no external encoders.<n>mname achieves a state-of-the-art FID of textbf2.40 within 400k steps, significantly outperforming comparable methods.
  arXiv  Detail & Related papers  (2025-12-09T14:39:26Z)
• CtrlDiff: Boosting Large Diffusion Language Models with Dynamic Block Prediction and Controllable Generation [13.250999667915254]
  Diffusion-based language models have emerged as a compelling alternative due to their powerful parallel generation capabilities and inherent editability.<n>We propose CtrlDiff, a dynamic and controllable semi-autoregressive framework that adaptively determines the size of each generation block based on local semantics.
  arXiv  Detail & Related papers  (2025-05-20T14:52:41Z)
• Constrained Discrete Diffusion [61.81569616239755]
  This paper introduces Constrained Discrete Diffusion (CDD), a novel integration of differentiable constraint optimization within the diffusion process.<n>CDD directly imposes constraints into the discrete diffusion sampling process, resulting in a training-free and effective approach.
  arXiv  Detail & Related papers  (2025-03-12T19:48:12Z)
• Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models [15.853201399662344]
  Diffusion language models offer unique benefits over autoregressive models.<n>They lag in likelihood modeling and are limited to fixed-length generation.<n>We introduce a class of block diffusion language models that interpolate between discrete denoising diffusion and autoregressive models.
  arXiv  Detail & Related papers  (2025-03-12T17:43:40Z)
• Generalized Interpolating Discrete Diffusion [65.74168524007484]
  Masked diffusion is a popular choice due to its simplicity and effectiveness.<n>We generalize a new family of general interpolating discrete diffusion (GIDD) which offers greater flexibility in the design of the noising processes.<n>Exploiting GIDD's flexibility, we explore a hybrid approach combining masking and uniform noise, leading to improved sample quality.
  arXiv  Detail & Related papers  (2025-03-06T14:30:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.