Adaptive Block Diffusion: Resolving Training-Inference Mismatch in Diffusion Language Models

• URL: http://arxiv.org/abs/2606.29275v1
• Date: Sun, 28 Jun 2026 08:45:00 GMT
• Title: Adaptive Block Diffusion: Resolving Training-Inference Mismatch in Diffusion Language Models
• Abstract summary: Diffusion Language Models (DLMs) are typically trained under fixed context structures, restricting denoising to predetermined token subsets.<n>We propose Adaptive Block Diffusion, which resolves this mismatch by optimizing denoising risk over a distribution of prefix-window configurations.<n>We show that generalization across decoding strategies is governed by the support of the training distribution, and that ABD guarantees denoising optimality for any inference policy whose configurations are covered during training.
• Score: 1.7626455083672685
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Diffusion Language Models (DLMs) are typically trained under fixed context structures, restricting denoising to predetermined token subsets. This creates a mismatch between training and inference, where models must operate over arbitrary configurations, leading to degradation off the training grid. We propose Adaptive Block Diffusion (ABD), which resolves this mismatch by optimizing denoising risk over a distribution of prefix-window configurations. By treating the configuration as a stochastic variable, ABD trains a single model over the full configuration space without architectural changes. We show that generalization across decoding strategies is governed by the support of the training distribution, and that ABD guarantees denoising optimality for any inference policy whose configurations are covered during training. Empirically, ABD exhibits structural invariance across decoding scales, avoiding off-grid collapse and recovering a monotonic relationship between block size and perplexity, while matching or outperforming fixed-block specialists at their target scales.

Related papers

• Discrete Diffusion Models: A Unified Framework from Tokenization to Generation [66.16696467962375]
  This work introduces a unified conceptual framework that views discrete diffusion models through the construction of the underlying discrete state space.<n>Within this framework, existing formulations, including transition-matrix, masking/absorbing-state, and score/ratio-based approaches, emerge as different instantiations of a common design space.
  arXiv  Detail & Related papers  (2026-07-15T04:23:22Z)
• Soft Specialists: $α$-Rényi Ensembles for Uncertainty-Aware LLM Post-Training [4.297453240868895]
  Existing training approaches for large language models learn a single set of parameters, based on large volumes of data.<n>We propose an $$-Rényi variational framework for learning distributions over post-training parameters.<n>We identify local stability criteria, demonstrating how model misspecification can make non-degenerate posterior spread locally favourable.
  arXiv  Detail & Related papers  (2026-05-26T22:44:58Z)
• Efficient Long-Context Modeling in Diffusion Language Models via Block Approximate Sparse Attention [58.484607892210015]
  Diffusion Language Models (DLMs) enable globally coherent, bidirectional, and controllable text generation.<n>Many existing block-sparse attention methods select blocks by fixed sampling patterns over the high-resolution attention space.<n>We propose the Block Approximate Sparse Attention framework (BA-Att) with block-wise pre-downsampled operation.
  arXiv  Detail & Related papers  (2026-05-19T12:01:13Z)
• Cluster-Aware Neural Collapse Prompt Tuning for Long-Tailed Generalization of Vision-Language Models [20.68004586002534]
  We propose cluster-aware neural collapse prompt tuning (CPT)<n>CPT enhances the discriminability of tail classes in prompt-tuned vision-language models without sacrificing generalization.<n>Experiments on 11 diverse datasets demonstrate that CPT outperforms SOTA methods.
  arXiv  Detail & Related papers  (2026-05-12T10:50:43Z)
• Self-Distilled Trajectory-Aware Boltzmann Modeling: Bridging the Training-Inference Discrepancy in Diffusion Language Models [65.89572755202245]
  Diffusion Language Models (DLMs) offer stronger global awareness and highly parallel generation.<n>Post-training DLMs with standard Negative Evidence Lower Bound (NELBO)-based supervised fine-tuning remains inefficient.<n>We propose a self-distilled trajectory-based post-training framework that aligns training with the easy-to-hard structure of inference.
  arXiv  Detail & Related papers  (2026-05-12T09:39:06Z)
• FB-NLL: A Feature-Based Approach to Tackle Noisy Labels in Personalized Federated Learning [2.750124853532831]
  We propose FB-NLL, a feature-centric framework that decouples user clustering from iterative training dynamics.<n>Our framework consistently outperforms state-of-the-art baselines in terms of average accuracy and performance stability.
  arXiv  Detail & Related papers  (2026-04-21T17:51:58Z)
• Too Correct to Learn: Reinforcement Learning on Saturated Reasoning Data [55.84428098924793]
  We propose Constrained Uniform Top-K Sampling (CUTS), a parameter-free decoding strategy enforcing structure-preserving exploration.<n>We integrate this into Mixed-CUTS, a training framework synergizing exploitative and exploratory rollouts to amplify intra-group advantage variance.<n> Notably, Mixed-CUTS improves Pass@1 accuracy on the challenging AIME25 benchmark by up to 15.1% over standard GRPO.
  arXiv  Detail & Related papers  (2026-04-20T16:43:28Z)
• From AR to Diffusion: Efficiently Adapting Large Language Models with Strictly Causal and Elastic Horizons [74.25767873168745]
  Diffusion models promise efficient parallel text generation but rely on bidirectional attention, creating a structural mismatch with pre-trained Autoregressive (AR) models.<n>We propose FLUID, a framework that efficiently adapts AR backbones to the diffusion paradigm.<n> Experiments demonstrate that FLUID achieves state-of-the-art performance while reducing training costs by orders of magnitude.
  arXiv  Detail & Related papers  (2026-04-11T13:18:27Z)
• AdaFuse: Adaptive Ensemble Decoding with Test-Time Scaling for LLMs [46.52320938421707]
  Inference-time ensembling provides a practical way to combine large language model capabilities without retraining.<n>We propose AdaFuse, an adaptive ensemble decoding framework that dynamically selects semantically appropriate fusion units during generation.<n>AdaFuse consistently outperforms strong ensemble baselines, achieving an average relative improvement of 6.88%.
  arXiv  Detail & Related papers  (2026-01-09T18:58:22Z)
• WavefrontDiffusion: Dynamic Decoding Schedule or Improved Reasoning [39.57157800995735]
  We present WavefrontDiffusion, a dynamic decoding approach that expands a wavefront of active tokens outward from finalized positions.<n>Across four benchmarks in reasoning and code generation, WavefrontDiffusion achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-11-22T07:33:00Z)
• Switchable Representation Learning Framework with Self-compatibility [50.48336074436792]
  We propose a Switchable representation learning Framework with Self-Compatibility (SFSC) SFSC generates a series of compatible sub-models with different capacities through one training process. SFSC achieves state-of-the-art performance on the evaluated datasets.
  arXiv  Detail & Related papers  (2022-06-16T16:46:32Z)
• Semi-supervised Domain Adaptive Structure Learning [72.01544419893628]
  Semi-supervised domain adaptation (SSDA) is a challenging problem requiring methods to overcome both 1) overfitting towards poorly annotated data and 2) distribution shift across domains. We introduce an adaptive structure learning method to regularize the cooperation of SSL and DA.
  arXiv  Detail & Related papers  (2021-12-12T06:11:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.