BlockGen: Flexible Blockwise Sequence Modeling with Hybrid Samplers

• URL: http://arxiv.org/abs/2606.02241v1
• Date: Mon, 01 Jun 2026 13:36:54 GMT
• Title: BlockGen: Flexible Blockwise Sequence Modeling with Hybrid Samplers
• Abstract summary: We introduce BlockGen, a blockwise sequence model that we instantiate with both masked and uniform diffusion.<n> BlockGen enables AR-informed predictor-corrector sampling (ARPC), which combines AR and diffusion predictions to re-generate unlikely tokens.<n>With block size $16$ on GSM8K, MDMs reach slightly higher accuracy than USDMs, and we observe a similar trend in Generative Perplexity on OpenWebText.
• Score: 12.083218729202963
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Is the uniform-state diffusion framework a more powerful paradigm for discrete diffusion? Recent studies indicate that this may be the case. In combination with predictor-corrector samplers, uniform-state diffusion models (USDMs) produce samples of higher-quality than masked diffusion models (MDMs), and USDMs equal or outperform MDMs in downstream tasks, even though they exhibit greater perplexity. Two issues remain unresolved. First, existing work compares uniform and masked diffusion with un-informed correctors that re-inject noise at random positions, rather than targeting tokens most likely to be wrong. Second, prior work compares full-sequence diffusion models, so we do not know whether the same conclusion holds when tokens are generated block by block. To address these issues, we introduce BlockGen, a blockwise sequence model that we instantiate with both masked and uniform diffusion. BlockGen trains on a mixture of block sizes and its likelihood interpolates between AR and pure diffusion more finely than models with a fixed block size. BlockGen enables AR-informed predictor-corrector sampling (ARPC), which combines AR and diffusion predictions to re-generate unlikely tokens without an auxiliary verifier. Under ancestral sampling, uniform outperforms masked in the block-by-block setting, especially in the few-step regime. Under ARPC, the gap closes and reverses at high NFE. With block size $16$ on GSM8K, MDMs reach slightly higher accuracy than USDMs, and we observe a similar trend in Generative Perplexity on OpenWebText. Find our code at https://github.com/jdeschena/blockgen.

Related papers

• Latent Block-Diffusion Temporal Point Processes: A Semi-Autoregressive Framework for Asynchronous Event Sequence Generation [48.64529006657015]
  We propose Latent Block-Diffusion Temporal Point Processes (LBDTPP) for high-quality and variable-length event sequence generation.<n>By sequentially generating blocks while simultaneously sampling events in each block, LBDTPP preserves the length flexibility of autoregressive TPPs.<n>LBDTPP outperforms state-of-the-art TPP baselines in both unconditional and conditional generation tasks.
  arXiv  Detail & Related papers  (2026-06-23T13:45:28Z)
• On the Trainability of Masked Diffusion Language Models via Blockwise Locality [58.92209096047332]
  Masked diffusion language models (MDMs) have recently emerged as a promising alternative to standard autoregressive large language models (AR-LLMs)<n>We study blockwise MDMs and compare them with AR-LLMs on three controlled tasks that stress different aspects of structured generation.<n>We find that standard random-masking MDMs fail to reliably learn linear regression, exhibit high variance training dynamics on graph path-finding, while outperforming AR-LLMs on Sudoku.
  arXiv  Detail & Related papers  (2026-04-27T17:44:26Z)
• Scaling Beyond Masked Diffusion Language Models [18.68471174706656]
  We present the first scaling law study of uniform-state and interpolating discrete diffusion methods.<n>We show that Masked diffusion models can be made approximately 12% more FLOPs-efficient when trained with a simple cross-entropy objective.
  arXiv  Detail & Related papers  (2026-02-16T18:54:47Z)
• Diffusion In Diffusion: Reclaiming Global Coherence in Semi-Autoregressive Diffusion [26.45111031153368]
  We propose a 'draft-then-refine' framework to overcome the irreversibility and myopia problems inherent in block diffusion models.<n>Our approach sets a new benchmark for discrete diffusion models on the OpenWebText dataset.
  arXiv  Detail & Related papers  (2026-01-20T05:00:26Z)
• From Next-Token to Next-Block: A Principled Adaptation Path for Diffusion LLMs [58.640039233470766]
  We show that principled AR-to-block-diffusion adaptation is an effective and compute-efficient alternative to training DLMs from scratch.<n> NBDiff-7B (Base and Instruct) could inherit the long-context modeling and reasoning capabilities, and achieve state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-12-07T10:28:21Z)
• Sequential Diffusion Language Models [110.06562906987052]
  Diffusion language models (DLMs) have strong theoretical efficiency but are limited by fixed-length decoding and incompatibility with key-value caches.<n>We introduce Next Sequence Prediction (NSP), which unifies next-token and next-block prediction.<n>We propose Sequential Diffusion Language Model (SDLM), which can retrofit pre-trained autoregressive language models (ALMs) at minimal cost.
  arXiv  Detail & Related papers  (2025-09-28T17:59:15Z)
• Blockwise SFT for Diffusion Language Models: Reconciling Bidirectional Attention and Autoregressive Decoding [60.06816407728172]
  Discrete diffusion language models have shown strong potential for text generation.<n>Standard supervised fine-tuning misaligns with semi-autoregressive inference.<n>We propose Blockwise SFT, which partitions responses into fixed-size blocks.
  arXiv  Detail & Related papers  (2025-08-27T02:49:33Z)
• Generalized Interpolating Discrete Diffusion [65.74168524007484]
  Masked diffusion is a popular choice due to its simplicity and effectiveness.<n>We generalize a new family of general interpolating discrete diffusion (GIDD) which offers greater flexibility in the design of the noising processes.<n>Exploiting GIDD's flexibility, we explore a hybrid approach combining masking and uniform noise, leading to improved sample quality.
  arXiv  Detail & Related papers  (2025-03-06T14:30:55Z)
• Remasking Discrete Diffusion Models with Inference-Time Scaling [21.362017006523086]
  We introduce the remasking diffusion model (ReMDM) sampler, a method that can be applied to pretrained masked diffusion models in a principled way.<n>Most interestingly, ReMDM endows discrete diffusion with a form of inference-time compute scaling.
  arXiv  Detail & Related papers  (2025-03-01T02:37:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.