Looped Diffusion Language Models

• URL: http://arxiv.org/abs/2605.26106v1
• Date: Mon, 25 May 2026 17:58:24 GMT
• Title: Looped Diffusion Language Models
• Abstract summary: Masked diffusion models (MDMs) have emerged as a promising alternative to autoregressive models for language modeling.<n>We show that selectively looping the early-middle transformer layers significantly improves both training efficiency and model performance in MDMs.
• Score: 47.8566079047557
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Masked diffusion models (MDMs) have emerged as a promising alternative to autoregressive models for language modeling, yet the effective design of transformer architectures for MDMs remains underexplored. In this paper, we show that selectively looping the early-middle transformer layers significantly improves both training efficiency and model performance in MDMs. We call this approach LoopMDM(Looped Masked Diffusion Model), which brings two key benefits: looping layers at training-time yields a depth-scaling effect without adding parameters, while varying the number of loops at inference-time enables flexible compute scaling. Despite the simplicity, the results are striking: across multiple pre-training corpora, LoopMDM matches the performance of same-size MDMs with up to 3.3 fewer training FLOPs, while its final performance outperforms them on various reasoning benchmarks, including up to 8.5 points on GSM8K. It even surpasses deeper non-looped MDMs trained with comparable per-step compute, indicating that selective looping is more effective than naive depth scaling. Furthermore, LoopMDM can scale inference-time compute by increasing the number of loops. Adaptively adjusting the number of loops throughout the sampling process further yields additional gains in compute efficiency while maintaining performance. Lastly, with attention analysis, we provide evidence that looping is effective in MDMs by promoting interactions among masked positions. Our code and weights will be publicly released.

Related papers

• Recursive Scaling in Masked Diffusion Models [67.93066069475753]
  Masked diffusion models (MDMs) have recently emerged as a promising paradigm for sequence generation.<n>We introduce Recursive Masked Diffusion Models (R-MDMs) to increase model depth without increasing parameter count.<n>Across structured generation tasks, including Sudoku and Countdown, we show that R-MDMs achieve substantially improved parameter efficiency.
  arXiv  Detail & Related papers  (2026-06-16T15:02:49Z)
• Structured Recurrent Mixers for Massively Parallelized Sequence Generation [0.0]
  We introduce the Structured Recurrent Mixer, an architecture that allows for conversion between a sequence parallel representation at train time and a recurrent representation at inference.<n>We show experimentally that this dual representation allows for greater training efficiency, higher input information capacity, and larger inference throughput and length when compared to other linear complexity models.
  arXiv  Detail & Related papers  (2026-05-09T05:07:55Z)
• Self-Rewarding Sequential Monte Carlo for Masked Diffusion Language Models [58.946955321428845]
  This work presents self-rewarding sequential Monte Carlo (SMC)<n>Our algorithm stems from the observation that most existing MDLMs rely on a confidence-based sampling strategy.<n>We introduce the trajectory-level confidence as a self-rewarding signal for assigning particle importance weights.
  arXiv  Detail & Related papers  (2026-02-02T09:21:45Z)
• Co-GRPO: Co-Optimized Group Relative Policy Optimization for Masked Diffusion Model [74.99242687133408]
  Masked Diffusion Models (MDMs) have shown promising potential across vision, language, and cross-modal generation.<n>We introduce Co-GRPO, which reformulates MDM generation as a unified Markov Decision Process (MDP) that jointly incorporates both the model and the inference schedule.
  arXiv  Detail & Related papers  (2025-12-25T12:06:04Z)
• Masked Diffusion Models are Secretly Learned-Order Autoregressive Models [21.17429712617749]
  We show that Masked Diffusion Models can identify and optimize for a decoding order during training.<n>We prove that the MDM objective decomposes precisely into a weighted auto-regressive losses over these orders.
  arXiv  Detail & Related papers  (2025-11-24T14:17:56Z)
• Sparse Training Scheme for Multimodal LLM [26.81140959413325]
  Multimodal Large Language Models (MLLMs) have demonstrated outstanding performance across a variety of domains.<n>We propose a novel training-efficient framework based on sparse representations, termed the Sparse Training Scheme (STS)<n>This scheme consists of two key components: the Visual Token, which reduces the information load by compressing visual tokens, and the Layer Dynamic Skipper, which mitigates the computational overhead by skipping unnecessary layers in the language model during both forward and backward passes.
  arXiv  Detail & Related papers  (2025-09-16T11:33:20Z)
• Variational Autoencoding Discrete Diffusion with Enhanced Dimensional Correlations Modeling [48.96034602889216]
  Variencoding Discrete Diffusion (VADD) is a novel framework that enhances discrete diffusion with latent variable modeling.<n>By introducing an auxiliary recognition model, VADD enables stable training via variational lower bounds and amortized inference over the training set.<n> Empirical results on 2D toy data, pixel-level image generation, and text generation demonstrate that VADD consistently outperforms MDM baselines.
  arXiv  Detail & Related papers  (2025-05-23T01:45:47Z)
• Dimple: Discrete Diffusion Multimodal Large Language Model with Parallel Decoding [53.82301522384719]
  We propose Dimple, the first Discrete Multimodal Large Language Model (DMLLM)<n>We design a novel training paradigm that combines an initial autoregressive phase with a subsequent diffusion phase.<n>Dimple-7B surpasses LLaVA- in performance by 3.9%, demonstrating that DMLLM can achieve performance comparable to that of autoregressive models.
  arXiv  Detail & Related papers  (2025-05-22T17:55:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.