Dynamic Chunking for Diffusion Language Models

• URL: http://arxiv.org/abs/2605.15676v1
• Date: Fri, 15 May 2026 06:56:05 GMT
• Title: Dynamic Chunking for Diffusion Language Models
• Abstract summary: Block discrete diffusion language models factorize a sequence autoregressively over fixed-size positional blocks.<n>We introduce textbfDynamic textbfChunking textbfDiffusion textbfModel (DCDM)<n>DCDM replaces positional blocks with content-defined semantic chunks.
• Score: 39.198939178122714
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Block discrete diffusion language models factorize a sequence autoregressively over fixed-size positional blocks, decoupling within-block parallel denoising from across-block conditioning. We argue that this rigid partition wastes structure already present in the sequence: blocks defined by position rather than by content separate semantically coherent tokens and group unrelated ones together. We introduce the \textbf{D}ynamic \textbf{C}hunking \textbf{D}iffusion \textbf{M}odel (DCDM), which replaces positional blocks with content-defined semantic chunks. At its core is Chunking Attention, a differentiable layer that routes tokens into $K$ clusters parameterized by learnable subspaces and shaped end-to-end by the diffusion objective. The resulting cluster assignments induce a chunk-causal attention mask under which a discrete diffusion denoiser factorizes the sequence likelihood autoregressively over semantic chunks, strictly generalizing block discrete diffusion. On downstream benchmarks at parameter scales up to 1.5B, DCDM consistently improves over both unstructured and positional-block diffusion baselines, with the advantage stable across scales and visible early in training.

Related papers

• Discrete Diffusion Models: A Unified Framework from Tokenization to Generation [66.16696467962375]
  This work introduces a unified conceptual framework that views discrete diffusion models through the construction of the underlying discrete state space.<n>Within this framework, existing formulations, including transition-matrix, masking/absorbing-state, and score/ratio-based approaches, emerge as different instantiations of a common design space.
  arXiv  Detail & Related papers  (2026-07-15T04:23:22Z)
• Set Diffusion: Interpolating Token Orderings Between Autoregression and Diffusion for Fast and Flexible Decoding [16.82407109143003]
  We present a new class of language models, set diffusion, comprised of (i) a likelihood parameterization that factorizes over flexible-position, flexible-length token sets and (ii) a set-causal diffusion architecture that supports KV cache updates after every inference step.<n>Set diffusion achieves better speed-quality tradeoffs on mathematical reasoning, summarization, and unconditional generation compared to prior diffusion language models.
  arXiv  Detail & Related papers  (2026-07-02T06:45:43Z)
• Adaptive Block Diffusion: Resolving Training-Inference Mismatch in Diffusion Language Models [1.7626455083672685]
  Diffusion Language Models (DLMs) are typically trained under fixed context structures, restricting denoising to predetermined token subsets.<n>We propose Adaptive Block Diffusion, which resolves this mismatch by optimizing denoising risk over a distribution of prefix-window configurations.<n>We show that generalization across decoding strategies is governed by the support of the training distribution, and that ABD guarantees denoising optimality for any inference policy whose configurations are covered during training.
  arXiv  Detail & Related papers  (2026-06-28T08:45:00Z)
• Prefix-Adaptive Block Diffusion for Efficient Document Recognition [52.15352911463151]
  Block Diffusion Models (BDMs) support parallel generation, flexible-length output, and KV caching, making them promising for efficient document parsing.<n>We propose the Prefix- Block Diffusion Model (PA-BDM) which replaces intra-block bidirectional denoising with causal denoising from prefix to suffix.<n> Experiments show that the 3B PA-BDM achieves higher recognition scores on several benchmarks and improves inference throughput by 71.6% over the 2.5B MinerU-Diffusion.
  arXiv  Detail & Related papers  (2026-05-16T07:50:13Z)
• Causal Autoregressive Diffusion Language Model [70.7353007255797]
  CARD reformulates the diffusion process within a strictly causal attention mask, enabling dense, per-token supervision in a single forward pass.<n>Our results demonstrate that CARD achieves ARM-level data efficiency while unlocking the latency benefits of parallel generation.
  arXiv  Detail & Related papers  (2026-01-29T17:38:29Z)
• On the Role of Discreteness in Diffusion LLMs [69.64854287505999]
  We revisit the view of diffusion process and language modeling, and outline five properties that separate diffusion mechanics from language-specific requirements.<n>We identify two central issues: (i) uniform corruption does not respect how information is distributed across positions, and (ii) token-wise marginal training cannot capture multi-token dependencies during parallel decoding.<n>These observations motivate diffusion processes that align more closely with the structure of text, and encourage future work toward more coherent diffusion language models.
  arXiv  Detail & Related papers  (2025-12-27T16:03:08Z)
• Continuously Augmented Discrete Diffusion model for Categorical Generative Modeling [87.34677262370924]
  Standard discrete diffusion models treat all unobserved states identically by mapping them to an absorbing [MASK] token.<n>This creates an 'information void' where semantic information that could be inferred from unmasked tokens is lost between denoising steps.<n>We introduce Continuously Augmented Discrete Diffusion, a framework that augments the discrete state space with a paired diffusion in a continuous latent space.
  arXiv  Detail & Related papers  (2025-10-01T18:00:56Z)
• Blockwise SFT for Diffusion Language Models: Reconciling Bidirectional Attention and Autoregressive Decoding [60.06816407728172]
  Discrete diffusion language models have shown strong potential for text generation.<n>Standard supervised fine-tuning misaligns with semi-autoregressive inference.<n>We propose Blockwise SFT, which partitions responses into fixed-size blocks.
  arXiv  Detail & Related papers  (2025-08-27T02:49:33Z)
• Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models [15.853201399662344]
  Diffusion language models offer unique benefits over autoregressive models.<n>They lag in likelihood modeling and are limited to fixed-length generation.<n>We introduce a class of block diffusion language models that interpolate between discrete denoising diffusion and autoregressive models.
  arXiv  Detail & Related papers  (2025-03-12T17:43:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.