When to Commit? Towards Variable-Size Self-Contained Blocks for Discrete Diffusion Language Models

• URL: http://arxiv.org/abs/2604.23994v1
• Date: Mon, 27 Apr 2026 03:21:07 GMT
• Title: When to Commit? Towards Variable-Size Self-Contained Blocks for Discrete Diffusion Language Models
• Abstract summary: We propose self-containedness as a principled criterion for block commitment.<n>A block is self-contained if its predictions remain consistent with Future-Aware (FA) or without No-Future (NF) access to future context.<n>We provide theoretical justification linking self-containedness to predictive consistency, and extensive experiments validate VSB's efficacy.
• Score: 36.08108046941572
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Discrete diffusion language models (dLLMs) enable parallel token updates with bidirectional attention, yet practical generation typically adopts blockwise semi-autoregressive decoding. This switch creates a training-inference mismatch: training denoises with full-sequence context, while inference commits tokens within a bounded block without future context. Therefore, decoding with fixed-size or heuristic-based blocks can lead to premature token commitments, as decisions are made without full access to future context that could alter those choices. Motivated by this, we propose self-containedness as a principled criterion for block commitment. A block is self-contained if its predictions remain consistent with Future-Aware (FA) or without No-Future (NF) access to future context, reframing block boundary selection as a test of self-containedness rather than a heuristic choice. Based on this principle, we introduce Variable-size Self-contained Blocks (VSB) for dLLMs. VSB scores and selects block boundaries using the divergence between token-level predictive distributions under NF and FA conditioning, which quantifies how predictions would change if future context were revealed. We provide theoretical justification linking self-containedness to predictive consistency, and extensive experiments validate VSB's efficacy over fixed-size and heuristic blockwise decoding.

Related papers

• DBLAST: Dependent Block Drafting for Stochastic Speculative Decoding [26.72775347772003]
  We show that the accepted draft length degrades as the entropy of the target sampling distribution increases.<n>We propose a dependent block drafter based on a low-rank latent mixture over token positions, complemented by an acceptance-oriented training objective.
  arXiv  Detail & Related papers  (2026-08-05T22:49:54Z)
• BlockBatch: Multi-Scale Consensus Decoding for Efficient Diffusion Language Model Inference [15.424211656458453]
  Diffusion language models generate text by iteratively denoising multiple token positions in parallel.<n>Small blocks preserve local conditioning but require many denoising steps, whereas large blocks expose more parallelism but can make premature commitments and accumulate cache error.<n>We propose BlockBatch, a training-free online inference framework that executes multiple block-size branches for the same request inside a batched forward pass.
  arXiv  Detail & Related papers  (2026-05-28T01:48:29Z)
• The Path Matters: Learning a Token-Commitment Policy for Diffusion Language Models [52.93186090124315]
  We argue that token commitment can be learned as a reusable trace-state policy.<n>We introduce TraceLock, a lightweight plug-in controller that instantiates this policy for a frozen diffusion language model.
  arXiv  Detail & Related papers  (2026-05-23T18:23:46Z)
• Prefix-Adaptive Block Diffusion for Efficient Document Recognition [52.15352911463151]
  Block Diffusion Models (BDMs) support parallel generation, flexible-length output, and KV caching, making them promising for efficient document parsing.<n>We propose the Prefix- Block Diffusion Model (PA-BDM) which replaces intra-block bidirectional denoising with causal denoising from prefix to suffix.<n> Experiments show that the 3B PA-BDM achieves higher recognition scores on several benchmarks and improves inference throughput by 71.6% over the 2.5B MinerU-Diffusion.
  arXiv  Detail & Related papers  (2026-05-16T07:50:13Z)
• Latent Thoughts Tuning: Bridging Context and Reasoning with Fused Information in Latent Tokens [13.653741247835091]
  Latent Thoughts Tuning (LT-Tuning) is a framework that redefines how latent thoughts are constructed and deployed.<n>We introduce a Context-Prediction-Fusion mechanism that jointly leveraging contextual hidden states and predictive semantic guidance.<n>Our method outperforms existing latent reasoning baselines, effectively mitigating feature collapse and achieving robust reasoning accuracy.
  arXiv  Detail & Related papers  (2026-02-10T19:19:10Z)
• CORE: Context-Robust Remasking for Diffusion Language Models [51.59514489363897]
  We propose Context-Robust Remasking (CORE), a training-free framework for inference-time revision.<n>Rather than trusting static token probabilities, CORE identifies context-brittle tokens by probing their sensitivity to targeted masked-context perturbations.<n>On LLaDA-8B-Base, CORE delivers consistent improvements across reasoning and code benchmarks, outperforming compute-matched baselines and improving MBPP by up to 9.2 percentage points.
  arXiv  Detail & Related papers  (2026-02-04T00:12:30Z)
• AdaBlock-dLLM: Semantic-Aware Diffusion LLM Inference via Adaptive Block Size [7.442463267121892]
  Diffusion-based large language models (dLLMs) are gaining attention for their inherent capacity for parallel decoding.<n>This paper presents the first systematic investigation challenging the fixed block size assumption in semi-AR decoding.<n>We introduce AdaBlock-dLLM, a training-free, plug-and-play scheduler that adaptively aligns block boundaries with semantic steps by adjusting block size during runtime.
  arXiv  Detail & Related papers  (2025-09-30T15:53:56Z)
• Blockwise SFT for Diffusion Language Models: Reconciling Bidirectional Attention and Autoregressive Decoding [60.06816407728172]
  Discrete diffusion language models have shown strong potential for text generation.<n>Standard supervised fine-tuning misaligns with semi-autoregressive inference.<n>We propose Blockwise SFT, which partitions responses into fixed-size blocks.
  arXiv  Detail & Related papers  (2025-08-27T02:49:33Z)
• Towards Universal Dense Blocking for Entity Resolution [49.06313308481536]
  We propose UniBlocker, a dense blocker that is pre-trained on a domain-independent, easily-obtainable corpus. By conducting domain-independent pre-training, UniBlocker can be adapted to various downstream blocking scenarios without requiring domain-specific fine-tuning. Our experiments show that the proposed UniBlocker, without any domain-specific learning, significantly outperforms previous self- and unsupervised dense blocking methods.
  arXiv  Detail & Related papers  (2024-04-23T08:39:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.