Dynamic Infilling Anchors for Format-Constrained Generation in Diffusion Large Language Models

• URL: http://arxiv.org/abs/2606.04535v1
• Date: Wed, 03 Jun 2026 07:18:23 GMT
• Title: Dynamic Infilling Anchors for Format-Constrained Generation in Diffusion Large Language Models
• Abstract summary: We propose Dynamic Infilling Anchors (DIA), a training-free method that dynamically estimates end-anchor positions to adjust generation length before iterative infilling.<n>DIA substantially improves format compliance and answer accuracy, achieving significant zero-shot gains on GSM8K and MATH.
• Score: 27.57238064480217
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Diffusion large language models (dLLMs) offer bidirectional attention and parallel generation, enabling them to exploit global context and naturally support format-constrained tasks like parseable JSON or reasoning templates. While straightforward fixed anchors can enforce such constraints, they often impose rigid spans, leading to truncated reasoning or redundant content. To overcome this, we propose Dynamic Infilling Anchors (DIA), a training-free method that dynamically estimates end-anchor positions to adjust generation length before iterative infilling. This flexible mechanism ensures structural correctness and semantic coherence, avoiding the inefficiencies of fixed-span methods. Experiments on reasoning benchmarks demonstrate that DIA substantially improves format compliance and answer accuracy, achieving significant zero-shot gains on GSM8K and MATH. These results establish DIA as a robust pathway toward reliable, structure-aware generation.

Related papers

• Weave of Formal Thought [51.56484100374058]
  We introduce Weave of Formal Thought (WoFT), a paradigm uniting rigorous syntactic validation with learned structural representations.<n>We present a latent-variable fine-tuning method training the language model to interleave non-terminal grammar symbols directly into generation.<n>For Python, fine-tuning StarCoder2-3B with our RWS objective reduces per-token cross-entropy by 14.3% relative to a text-only SFT baseline.
  arXiv  Detail & Related papers  (2026-06-24T15:58:11Z)
• Self-Distilled Trajectory-Aware Boltzmann Modeling: Bridging the Training-Inference Discrepancy in Diffusion Language Models [65.89572755202245]
  Diffusion Language Models (DLMs) offer stronger global awareness and highly parallel generation.<n>Post-training DLMs with standard Negative Evidence Lower Bound (NELBO)-based supervised fine-tuning remains inefficient.<n>We propose a self-distilled trajectory-based post-training framework that aligns training with the easy-to-hard structure of inference.
  arXiv  Detail & Related papers  (2026-05-12T09:39:06Z)
• Dystruct: Dynamically Structured Diffusion Language Model Decoding via Bayesian Inference [51.12849550784653]
  Diffusion language models (DLMs) have emerged as a promising alternative to autoregressive models.<n>Most existing DLMs rely on a fixed generation length specified prior to decoding, which restricts their flexibility in real-world applications.<n>We propose a training-free, Bayesian structured decoding framework that formulates flexible-length generation as a dynamic structural inference problem.
  arXiv  Detail & Related papers  (2026-05-10T23:51:14Z)
• From AR to Diffusion: Efficiently Adapting Large Language Models with Strictly Causal and Elastic Horizons [74.25767873168745]
  Diffusion models promise efficient parallel text generation but rely on bidirectional attention, creating a structural mismatch with pre-trained Autoregressive (AR) models.<n>We propose FLUID, a framework that efficiently adapts AR backbones to the diffusion paradigm.<n> Experiments demonstrate that FLUID achieves state-of-the-art performance while reducing training costs by orders of magnitude.
  arXiv  Detail & Related papers  (2026-04-11T13:18:27Z)
• FMBench: Adaptive Large Language Model Output Formatting [49.52930069696333]
  We present FMBench, a benchmark for adaptive Markdown output formatting.<n>Experiments on two model families show that SFT consistently improves semantic alignment.<n>Results also reveal an inherent trade-off between semantic and structural objectives.
  arXiv  Detail & Related papers  (2026-02-06T04:42:06Z)
• Lookahead-then-Verify: Reliable Constrained Decoding for Diffusion LLMs under Context-Free Grammars [17.13122301190815]
  We present LAVE, a constrained decoding approach specifically designed for dLLMs.<n>Our approach leverages a key property of dLLMs, namely their ability to predict token distributions for all positions in parallel during each forward pass.<n>Extensive experiments across four widely used dLLMs and three representative benchmarks demonstrate that LAVE consistently outperforms existing baselines and achieves substantial improvements in syntactic correctness, while incurring negligible runtime overhead.
  arXiv  Detail & Related papers  (2026-01-31T08:58:15Z)
• WavefrontDiffusion: Dynamic Decoding Schedule or Improved Reasoning [39.57157800995735]
  We present WavefrontDiffusion, a dynamic decoding approach that expands a wavefront of active tokens outward from finalized positions.<n>Across four benchmarks in reasoning and code generation, WavefrontDiffusion achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-11-22T07:33:00Z)
• Unlocking the Potential of Diffusion Language Models through Template Infilling [33.69224085914102]
  Diffusion Language Models (DLMs) have emerged as a promising alternative to Autoregressive Language Models.<n>We propose Template Infilling (TI), a tailored conditioning methodology for DLMs' generation process.<n>We show that TI provides additional advantages in multi-token generation settings, enabling effective speedup while maintaining generation quality.
  arXiv  Detail & Related papers  (2025-10-13T12:33:41Z)
• Beyond Fixed: Training-Free Variable-Length Denoising for Diffusion Large Language Models [74.15250326312179]
  Diffusion Large Language Models offer efficient parallel generation and capable global modeling.<n>The dominant application ofDLLMs is hindered by the need for a statically predefined generation length.<n>We introduce DAEDAL, a novel training-free denoising strategy that enables Dynamic Adaptive Length Expansion.
  arXiv  Detail & Related papers  (2025-08-01T17:56:07Z)
• CtrlDiff: Boosting Large Diffusion Language Models with Dynamic Block Prediction and Controllable Generation [13.250999667915254]
  Diffusion-based language models have emerged as a compelling alternative due to their powerful parallel generation capabilities and inherent editability.<n>We propose CtrlDiff, a dynamic and controllable semi-autoregressive framework that adaptively determines the size of each generation block based on local semantics.
  arXiv  Detail & Related papers  (2025-05-20T14:52:41Z)
• Improve Variational Autoencoder for Text Generationwith Discrete Latent Bottleneck [52.08901549360262]
  Variational autoencoders (VAEs) are essential tools in end-to-end representation learning. VAEs tend to ignore latent variables with a strong auto-regressive decoder. We propose a principled approach to enforce an implicit latent feature matching in a more compact latent space.
  arXiv  Detail & Related papers  (2020-04-22T14:41:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.