Dystruct: Dynamically Structured Diffusion Language Model Decoding via Bayesian Inference

• URL: http://arxiv.org/abs/2605.09820v1
• Date: Sun, 10 May 2026 23:51:14 GMT
• Title: Dystruct: Dynamically Structured Diffusion Language Model Decoding via Bayesian Inference
• Abstract summary: Diffusion language models (DLMs) have emerged as a promising alternative to autoregressive models.<n>Most existing DLMs rely on a fixed generation length specified prior to decoding, which restricts their flexibility in real-world applications.<n>We propose a training-free, Bayesian structured decoding framework that formulates flexible-length generation as a dynamic structural inference problem.
• Score: 51.12849550784653
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Diffusion language models (DLMs) have recently emerged as a promising alternative to autoregressive models, primarily due to their ability to enable parallel decoding. Despite this advantage, most existing DLMs rely on a fixed generation length specified prior to decoding, which restricts their flexibility in real-world applications. While a few recent works attempt to support flexible-length generation, they typically suffer from notable limitations: some require costly retraining to accommodate variable-length outputs, while others depend solely on local confidence signals during decoding. Such local criteria fail to capture the evolving structure of the sequence, often resulting in suboptimal generation quality. In this paper, we propose a training-free, Bayesian structured decoding framework that formulates flexible-length generation as a dynamic structural inference problem. Our approach formulates flexible-length generation as a dynamic structural inference problem, jointly computing the expansion length, the block boundaries, and the decoding schedule. At each window expansion step, the method integrates local uncertainty with structural signals via a unified mechanism that supports dynamic structured generation, including both flexible block expansion and block organization, while maintaining coherence. Extensive experiments across multiple benchmarks demonstrate that our approach significantly improves generation quality and flexibility over existing fixed-length and flexible-length baselines. These results highlight the advantage of Bayesian structured decoding for diffusion language model, providing a principled and efficient solution for structured text generation.

Related papers

• IS-CoT: Breaking the Long-form Generation Collapse via Interleaved Structural Thinking [50.381430711141995]
  We introduce the Interleaved Structural Chain-of-Thought (IS-CoT) framework.<n>IS-CoT embeds a dynamic Plan-Write-Reflect cycle into the generation process.<n>We construct a high-quality dataset of interleaved reasoning traces via a multi-teacher pipeline and train IS-Writer-8B.
  arXiv  Detail & Related papers  (2026-06-08T16:31:00Z)
• Dynamic Infilling Anchors for Format-Constrained Generation in Diffusion Large Language Models [27.57238064480217]
  We propose Dynamic Infilling Anchors (DIA), a training-free method that dynamically estimates end-anchor positions to adjust generation length before iterative infilling.<n>DIA substantially improves format compliance and answer accuracy, achieving significant zero-shot gains on GSM8K and MATH.
  arXiv  Detail & Related papers  (2026-06-03T07:18:23Z)
• Continuous Latent Diffusion Language Model [48.974403879186916]
  Large language models have achieved remarkable success under the autoregressive paradigm.<n>Existing alternatives still struggle to jointly achieve generation efficiency, scalable representation learning, and effective global semantic modeling.<n>We propose Cola DLM, a hierarchical latent diffusion language model that frames text generation through hierarchical information decomposition.
  arXiv  Detail & Related papers  (2026-05-07T16:44:56Z)
• Flow of Spans: Generalizing Language Models to Dynamic Span-Vocabulary via GFlowNets [54.06320619464273]
  Flow of SpanS (FOSS) is a principled GFlowNets framework for span generation.<n>FOSS constructs a dynamic span vocabulary by segmenting the retrieved text flexibly.<n>With specialized reward models, FoSS generates diverse, high-quality text.
  arXiv  Detail & Related papers  (2026-02-11T07:17:41Z)
• AdaFuse: Adaptive Ensemble Decoding with Test-Time Scaling for LLMs [46.52320938421707]
  Inference-time ensembling provides a practical way to combine large language model capabilities without retraining.<n>We propose AdaFuse, an adaptive ensemble decoding framework that dynamically selects semantically appropriate fusion units during generation.<n>AdaFuse consistently outperforms strong ensemble baselines, achieving an average relative improvement of 6.88%.
  arXiv  Detail & Related papers  (2026-01-09T18:58:22Z)
• Unlocking the Potential of Diffusion Language Models through Template Infilling [33.69224085914102]
  Diffusion Language Models (DLMs) have emerged as a promising alternative to Autoregressive Language Models.<n>We propose Template Infilling (TI), a tailored conditioning methodology for DLMs' generation process.<n>We show that TI provides additional advantages in multi-token generation settings, enabling effective speedup while maintaining generation quality.
  arXiv  Detail & Related papers  (2025-10-13T12:33:41Z)
• Beyond Fixed: Training-Free Variable-Length Denoising for Diffusion Large Language Models [74.15250326312179]
  Diffusion Large Language Models offer efficient parallel generation and capable global modeling.<n>The dominant application ofDLLMs is hindered by the need for a statically predefined generation length.<n>We introduce DAEDAL, a novel training-free denoising strategy that enables Dynamic Adaptive Length Expansion.
  arXiv  Detail & Related papers  (2025-08-01T17:56:07Z)
• LoViC: Efficient Long Video Generation with Context Compression [68.22069741704158]
  We introduce LoViC, a DiT-based framework trained on million-scale open-domain videos.<n>At the core of our approach is FlexFormer, an expressive autoencoder that jointly compresses video and text into unified latent representations.
  arXiv  Detail & Related papers  (2025-07-17T09:46:43Z)
• CtrlDiff: Boosting Large Diffusion Language Models with Dynamic Block Prediction and Controllable Generation [13.250999667915254]
  Diffusion-based language models have emerged as a compelling alternative due to their powerful parallel generation capabilities and inherent editability.<n>We propose CtrlDiff, a dynamic and controllable semi-autoregressive framework that adaptively determines the size of each generation block based on local semantics.
  arXiv  Detail & Related papers  (2025-05-20T14:52:41Z)
• Speculative Diffusion Decoding: Accelerating Language Generation through Diffusion [55.0194604505437]
  Speculative decoding has emerged as a widely adopted method to accelerate large language model inference.<n>This paper proposes an adaptation of speculative decoding which uses discrete diffusion models to generate draft sequences.
  arXiv  Detail & Related papers  (2024-08-10T21:24:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.