On the Trainability of Masked Diffusion Language Models via Blockwise Locality

• URL: http://arxiv.org/abs/2604.24832v1
• Date: Mon, 27 Apr 2026 17:44:26 GMT
• Title: On the Trainability of Masked Diffusion Language Models via Blockwise Locality
• Abstract summary: Masked diffusion language models (MDMs) have recently emerged as a promising alternative to standard autoregressive large language models (AR-LLMs)<n>We study blockwise MDMs and compare them with AR-LLMs on three controlled tasks that stress different aspects of structured generation.<n>We find that standard random-masking MDMs fail to reliably learn linear regression, exhibit high variance training dynamics on graph path-finding, while outperforming AR-LLMs on Sudoku.
• Score: 58.92209096047332
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Masked diffusion language models (MDMs) have recently emerged as a promising alternative to standard autoregressive large language models (AR-LLMs), yet their optimization can be substantially less stable. We study blockwise MDMs and compare them with AR-LLMs on three controlled tasks that stress different aspects of structured generation: in-context linear regression, graph path-finding, and Sudoku solving. We find that standard random-masking MDMs fail to reliably learn linear regression, exhibit high variance training dynamics on graph path-finding, while outperforming AR-LLMs on Sudoku. To mitigate these instabilities, we propose two locality aware blockwise models, namely Jigsaw and Scatter, that inject left-to-right inductive bias by enforcing autoregressive locality within blocks while preserving iterative refinement at the block level. Empirically, Jigsaw matches AR-LLM stability on linear regression and remains strong on Sudoku, while Scatter retains diffusion's planning advantage on path-finding. Our results indicate that standard random-masking MDMs, even with blockwise variants, may be a suboptimal instantiation of diffusion LMs for ordered generation, motivating models beyond random masking.

Related papers

• Uncertainty Quantification for Large Language Diffusion Models [35.87984313223733]
  Large Language Diffusion Models (LLDMs) are emerging as an alternative to autoregressive models.<n>Existing uncertainty quantification (UQ) methods assume autoregressive factorization or use expensive repeated sampling.<n>We present the first systematic study of UQ for LLDMs and propose lightweight, zero-shot uncertainty signals.
  arXiv  Detail & Related papers  (2026-05-14T08:39:56Z)
• Understanding and Accelerating the Training of Masked Diffusion Language Models [104.60946175023008]
  Masked diffusion models (MDMs) have emerged as a promising alternative to autoregressive models (ARMs) for language modeling.<n>We show how to accelerate standard MDM training while maintaining its final performance.<n>We also show faster improvements in generative perplexity, zero-shot perplexity, and downstream task performance on various benchmarks.
  arXiv  Detail & Related papers  (2026-05-13T05:29:38Z)
• The Efficiency Gap in Byte Modeling [101.18202046105718]
  Two alternative paradigms have challenged the use of subword tokenization and autoregressive ordering.<n>We show that removing these structural priors incurs a significant computational cost.<n>Our findings suggest that future modality-agnostic designs must incorporate alternative structural biases to maintain viable scaling trajectories.
  arXiv  Detail & Related papers  (2026-05-13T03:03:30Z)
• Discrete Tilt Matching [10.240234517598378]
  Masked diffusion large language models (dLLMs) are a promising alternative to autoregressive generation.<n>We derive Discrete Tilt Matching (DTM), a likelihood-free method that recasts dLLM fine-tuning as state-level matching of local unmasking posteriors.<n>DTM yields strong gains on Sudoku and Countdown while remaining competitive on MATH500 and GSM8K.
  arXiv  Detail & Related papers  (2026-04-20T18:43:37Z)
• MAR-GRPO: Stabilized GRPO for AR-diffusion Hybrid Image Generation [24.618644100413018]
  Reinforcement learning (RL) has been successfully applied to autoregressive (AR) and diffusion models.<n> extending RL to hybrid AR-diffusion frameworks remains challenging due to interleaved inference and noisy log-probability estimation.<n>In this work, we study masked autoregressive models (MAR) and show that the diffusion head plays a critical role in training dynamics.
  arXiv  Detail & Related papers  (2026-04-08T11:30:35Z)
• Unifying Masked Diffusion Models with Various Generation Orders and Beyond [56.70289720766803]
  Masked diffusion models (MDMs) are a potential alternative to autoregressive models (ARMs) for language generation.<n>We propose order-expressive masked diffusion model (OeMDM) for a broad class of diffusion generative processes.<n>We introduce learnable-order masked diffusion model (LoMDM) which jointly learns the generation ordering and diffusion backbone.
  arXiv  Detail & Related papers  (2026-02-02T13:54:32Z)
• Self-Rewarding Sequential Monte Carlo for Masked Diffusion Language Models [58.946955321428845]
  This work presents self-rewarding sequential Monte Carlo (SMC)<n>Our algorithm stems from the observation that most existing MDLMs rely on a confidence-based sampling strategy.<n>We introduce the trajectory-level confidence as a self-rewarding signal for assigning particle importance weights.
  arXiv  Detail & Related papers  (2026-02-02T09:21:45Z)
• Causal Autoregressive Diffusion Language Model [70.7353007255797]
  CARD reformulates the diffusion process within a strictly causal attention mask, enabling dense, per-token supervision in a single forward pass.<n>Our results demonstrate that CARD achieves ARM-level data efficiency while unlocking the latency benefits of parallel generation.
  arXiv  Detail & Related papers  (2026-01-29T17:38:29Z)
• MDPO: Overcoming the Training-Inference Divide of Masked Diffusion Language Models [28.79185891706149]
  Diffusion language models suffer from a key discrepancy between training and inference.<n>We propose a novel Masked Diffusion Policy Optimization (MDPO) to exploit the Markov property diffusion.<n>Our findings establish great potential for investigating the discrepancy between pre-training and inference of MDLMs.
  arXiv  Detail & Related papers  (2025-08-18T17:58:13Z)
• Remasking Discrete Diffusion Models with Inference-Time Scaling [21.362017006523086]
  We introduce the remasking diffusion model (ReMDM) sampler, a method that can be applied to pretrained masked diffusion models in a principled way.<n>Most interestingly, ReMDM endows discrete diffusion with a form of inference-time compute scaling.
  arXiv  Detail & Related papers  (2025-03-01T02:37:51Z)
• Beyond Autoregression: Discrete Diffusion for Complex Reasoning and Planning [89.96284387376119]
  We show how diffusion models learn difficult subgoals that elude autoregressive approaches.<n>We propose Multi-Granularity Diffusion Modeling (MGDM), which prioritizes subgoals based on difficulty during learning.<n>MGDM significantly outperforms autoregressive models without using search techniques.
  arXiv  Detail & Related papers  (2024-10-18T03:48:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.