Locally Coherent Parallel Decoding in Diffusion Language Models

• URL: http://arxiv.org/abs/2603.20216v1
• Date: Tue, 03 Mar 2026 09:56:53 GMT
• Title: Locally Coherent Parallel Decoding in Diffusion Language Models
• Abstract summary: Diffusion language models (DLMs) offer sub-linear generation latency and bidirectional capabilities.<n>Standard DLMs sample tokens independently from conditional marginal distributions.<n>We introduce CoDiLA, a method that reconciles parallel sampling with local dependency modeling.
• Score: 6.620088179445404
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Diffusion language models (DLMs) have emerged as a promising alternative to autoregressive (AR) models, offering sub-linear generation latency and bidirectional capabilities that are particularly appealing for code generation and editing. Achieving sub-linear latency in discrete DLMs requires predicting multiple tokens in parallel. However, standard DLMs sample tokens independently from conditional marginal distributions, failing to capture the joint dependencies among concurrently generated tokens. As a result, they often lead to syntactic inconsistencies and break multi-token structures. In this work, we introduce CoDiLA (Coherent Diffusion with Local Autoregression), a method that reconciles parallel sampling with local dependency modeling. Rather than forcing the DLM to resolve fine-grained syntax, CoDiLA delegates local decoding to a small, auxiliary AR model operating on the diffusion latents. This design allows for parallel block generation while ensuring sequential validity within each block and maintaining core DLM capabilities, including bidirectional modeling across blocks. We demonstrate that using a highly compact auxiliary AR model (e.g., 0.6B parameters) effectively eliminates coherence artifacts, establishing a new Pareto frontier for accuracy and speed in code generation benchmarks.

Related papers

• AURORA-LM: Autoencoding Unified Representation for Continuous-Latent Diffusion Language Modeling [60.04959047453115]
  AURORA-LM is a continuous-latent language model that separates the construction of a decodable text representation from the modeling of its distribution.<n>AURORA-LM achieves the strongest performance among evaluated continuous and diffusion-based language models on OpenWebText free generation and XSum summarization.
  arXiv  Detail & Related papers  (2026-08-03T17:59:50Z)
• Rethinking the Generation Order of Block Diffusion Language Models [4.187801730089504]
  Diffusion language models enable flexible arbitrary-order generation, but existing sampling methods are mostly designed for early masked diffusion models (MDMs)<n>We propose Parallel Autoregressive Decoding (PARD), a simple training-free sampling method that preserves left-to-right unmasking structure while allowing parallel token commitment.
  arXiv  Detail & Related papers  (2026-07-27T11:49:50Z)
• Mean-Field Parallel Decoding for Discrete Diffusion Language Models [47.086589631406035]
  We introduce a training-free decoding framework that coordinates parallel updates.<n>A variational relaxation yields a simple fixed-point update that suppresses conflicting simultaneous commitments.<n> Experiments on reasoning and code-generation benchmarks show consistent improvements in the quality-latency trade-off.
  arXiv  Detail & Related papers  (2026-06-14T13:17:58Z)
• SimSD: Simple Speculative Decoding in Diffusion Language Models [61.33773959352141]
  Diffusion large language models (dLLMs) offer faster inference through parallel or blockwise decoding.<n>Their masked language modeling formulation remains incompatible with standard token-level speculative decoding.<n>We propose a speculative decoding algorithm for dLLMs, named SimSD, that equips dLLMs with temporally valid token-level contexts.<n>Our method achieves up to 7.46x higher decoding throughput while maintaining and even improving average generation quality.
  arXiv  Detail & Related papers  (2026-06-01T17:46:46Z)
• How to Train Your Latent Diffusion Language Model Jointly With the Latent Space [76.9057986588963]
  Latent diffusion models offer an attractive alternative to discrete diffusion for non-autoregressive text generation.<n>We present the Latent Diffusion Language Model (LDLM), in which the latent encoder, diffusion model, and decoder are trained jointly.<n>On OpenWebText and LM1B, LDLM achieves better generation performance than existing discrete and continuous diffusion language models.
  arXiv  Detail & Related papers  (2026-05-08T16:05:19Z)
• CRoCoDiL: Continuous and Robust Conditioned Diffusion for Language [21.764828393078886]
  Masked Diffusion Models (MDMs) provide an efficient non-causal alternative to autoregressive generation.<n>We address these limitations by shifting the diffusion process into a continuous sentence-level semantic space.
  arXiv  Detail & Related papers  (2026-03-02T12:50:55Z)
• CDLM: Consistency Diffusion Language Models For Faster Sampling [54.886467592798]
  Diffusion Language Models (DLMs) offer a promising parallel generation paradigm but suffer from slow inference.<n>We introduce CDLM, a training-based acceleration method that simultaneously tackles both bottlenecks.<n>Experiments show CDLM achieves 3.6x-14.5x lower latency while maintaining competitive accuracy on math and coding tasks.
  arXiv  Detail & Related papers  (2025-11-24T16:21:25Z)
• Accelerating Diffusion LLMs via Adaptive Parallel Decoding [60.407727995313074]
  We introduce adaptive parallel decoding (APD), a novel method that dynamically adjusts the number of tokens sampled in parallel.<n>APD provides markedly higher throughput with minimal quality degradations on downstream benchmarks.
  arXiv  Detail & Related papers  (2025-05-31T06:10:10Z)
• Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding [51.711605076319216]
  Diffusion-based large language models (Diffusion LLMs) have shown promise for non-autoregressive text generation with parallel decoding capabilities.<n>We introduce a novel block-wise approximate KV Cache mechanism tailored for bidirectional diffusion models, enabling cache reuse with negligible performance drop.<n>We propose a confidence-aware parallel decoding strategy that selectively decodes tokens exceeding a confidence threshold, mitigating dependency violations and maintaining generation quality.
  arXiv  Detail & Related papers  (2025-05-28T17:39:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.