Infinite Mask Diffusion for Few-Step Distillation

• URL: http://arxiv.org/abs/2605.10518v1
• Date: Mon, 11 May 2026 13:07:03 GMT
• Title: Infinite Mask Diffusion for Few-Step Distillation
• Abstract summary: Masked Diffusion Models (MDMs) have emerged as a promising alternative to autoregressive models in language modeling.<n>We propose the Infinite Mask Diffusion Model (IMDM) to mitigate the theoretical bound while inheriting the benefits of MDMs.<n>IMDM surpasses existing few-step distillation methods at small step counts on LM1B and OpenWebText.
• Score: 16.340935269530842
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Masked Diffusion Models (MDMs) have emerged as a promising alternative to autoregressive models in language modeling, offering the advantages of parallel decoding and bidirectional context processing within a simple yet effective framework. Specifically, their explicit distinction between masked tokens and data underlies their simple framework and effective conditional generation. However, MDMs typically require many sampling iterations due to factorization errors stemming from simultaneous token updates. We observe that a theoretical lower bound of the factorization error exists, which standard MDMs cannot reduce due to their use of a deterministic single-state mask. In this paper, we propose the Infinite Mask Diffusion Model (IMDM), which introduces a stochastic infinite-state mask to mitigate the theoretical bound while directly inheriting the benefits of MDMs, including the compatibility with pre-trained weights. We empirically demonstrate that MDM fails to perform few-step generation even in a simple synthetic task due to the factorization error bound, whereas IMDM can find an efficient solution for the same task. Finally, when equipped with appropriate distillation methods, IMDM surpasses existing few-step distillation methods at small step counts on LM1B and OpenWebText. Code is available at https://Ugness.github.io/official_imdm.

Related papers

• Multi-Mask Diffusion Language Models for Few-Step Generation [67.14463385328135]
  Masked diffusion models (MDMs) are a promising family of language generators.<n>Recent few-step alternatives based on uniform-state diffusion avoid this degeneracy.<n>We propose a multi-mask diffusion model (MultiMDM) that preserves the masking structure towards few-step generation.
  arXiv  Detail & Related papers  (2026-07-22T02:35:40Z)
• Masked Language Flow Models [10.204151640924454]
  Flow Language Models (FLMs) sidestep this limitation by learning a continuous flow that transports noise toward clean sequences.<n>We introduce Masked Language Flow Models (MLFMs) which incorporate masking into FLMs using a continuous interpolant to bridge partially masked and clean sequences.<n>We find, for the first time, that flow-based language models can be scaled to solve downstream reasoning and instruction-following tasks.
  arXiv  Detail & Related papers  (2026-06-26T00:16:40Z)
• SimSD: Simple Speculative Decoding in Diffusion Language Models [61.33773959352141]
  Diffusion large language models (dLLMs) offer faster inference through parallel or blockwise decoding.<n>Their masked language modeling formulation remains incompatible with standard token-level speculative decoding.<n>We propose a speculative decoding algorithm for dLLMs, named SimSD, that equips dLLMs with temporally valid token-level contexts.<n>Our method achieves up to 7.46x higher decoding throughput while maintaining and even improving average generation quality.
  arXiv  Detail & Related papers  (2026-06-01T17:46:46Z)
• On the Trainability of Masked Diffusion Language Models via Blockwise Locality [58.92209096047332]
  Masked diffusion language models (MDMs) have recently emerged as a promising alternative to standard autoregressive large language models (AR-LLMs)<n>We study blockwise MDMs and compare them with AR-LLMs on three controlled tasks that stress different aspects of structured generation.<n>We find that standard random-masking MDMs fail to reliably learn linear regression, exhibit high variance training dynamics on graph path-finding, while outperforming AR-LLMs on Sudoku.
  arXiv  Detail & Related papers  (2026-04-27T17:44:26Z)
• VocalNet-MDM: Accelerating Streaming Speech LLM via Self-Distilled Masked Diffusion Modeling [31.58493743596625]
  Masked Diffusion Modeling(MDM) is a non-autoregressive paradigm for speech LLMs.<n> VocalNet-MDM is trained on a limited scale of only 6K hours of speech data.<n>It maintains competitive recognition accuracy while achieving state-of-the-art text quality and speech naturalness.
  arXiv  Detail & Related papers  (2026-02-09T12:52:59Z)
• Sparse-LaViDa: Sparse Multimodal Discrete Diffusion Language Models [63.50827603618498]
  We propose Sparse-LaViDa, a modeling framework that truncates unnecessary masked tokens at each inference step to accelerate MDM sampling.<n>Built upon the state-of-the-art unified MDM LaViDa-O, Sparse-LaViDa achieves up to a 2x speedup across diverse tasks.
  arXiv  Detail & Related papers  (2025-12-16T02:06:06Z)
• On the Reasoning Abilities of Masked Diffusion Language Models [34.937359931224265]
  Masked diffusion models (MDMs) for text offer a compelling alternative to traditional autoregressive language models.<n>We characterize what types of reasoning problems MDMs can provably solve and how efficiently.
  arXiv  Detail & Related papers  (2025-10-15T03:29:26Z)
• Fine-Tuning Masked Diffusion for Provable Self-Correction [28.338622227684453]
  Masked Diffusion Models (MDMs) have emerged as a promising approach for generative modeling in discrete spaces.<n>We introduce PRISM--Plug-in Remasking for Inference-time Self-correction of Masked Diffusions.
  arXiv  Detail & Related papers  (2025-10-01T19:15:25Z)
• Any-Order Flexible Length Masked Diffusion [53.89217188409148]
  Masked diffusion models (MDMs) have recently emerged as a promising alternative to autoregressive models over discrete domains.<n>We introduce Flexible Masked Diffusion Models (FlexMDMs), a discrete diffusion paradigm that simultaneously can model sequences of flexible length.<n>We show that FlexMDMs match MDMs in perplexity while modeling length statistics with much higher fidelity.
  arXiv  Detail & Related papers  (2025-08-31T23:34:53Z)
• Beyond Masked and Unmasked: Discrete Diffusion Models via Partial Masking [28.55159825491572]
  Masked diffusion models (MDM) are powerful generative models for discrete data that generate samples by progressively unmasking tokens in a sequence.<n>We propose the Partial masking scheme (Prime), which augments MDM by allowing tokens to take intermediate states between the masked and unmasked states.<n>Our method demonstrates superior performance across a diverse set of generative modeling tasks.
  arXiv  Detail & Related papers  (2025-05-24T04:16:40Z)
• Unsupervised Modality Adaptation with Text-to-Image Diffusion Models for Semantic Segmentation [54.96563068182733]
  We propose Modality Adaptation with text-to-image Diffusion Models (MADM) for semantic segmentation task. MADM utilizes text-to-image diffusion models pre-trained on extensive image-text pairs to enhance the model's cross-modality capabilities. We show that MADM achieves state-of-the-art adaptation performance across various modality tasks, including images to depth, infrared, and event modalities.
  arXiv  Detail & Related papers  (2024-10-29T03:49:40Z)
• Steering Masked Discrete Diffusion Models via Discrete Denoising Posterior Prediction [88.65168366064061]
  We introduce Discrete Denoising Posterior Prediction (DDPP), a novel framework that casts the task of steering pre-trained MDMs as a problem of probabilistic inference. Our framework leads to a family of three novel objectives that are all simulation-free, and thus scalable. We substantiate our designs via wet-lab validation, where we observe transient expression of reward-optimized protein sequences.
  arXiv  Detail & Related papers  (2024-10-10T17:18:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.