MDM-Prime-v2: Binary Encoding and Index Shuffling Enable Compute-optimal Scaling of Diffusion Language Models

• URL: http://arxiv.org/abs/2603.16077v1
• Date: Tue, 17 Mar 2026 02:54:16 GMT
• Title: MDM-Prime-v2: Binary Encoding and Index Shuffling Enable Compute-optimal Scaling of Diffusion Language Models
• Abstract summary: Masked diffusion models (MDM) exhibit superior generalization when learned using a Partial masking scheme (Prime)<n>We develop MDM-Prime-v2, a masked diffusion language model which incorporates Binary sense and Shuffling.<n>In compute-optimal comparisons, MDM-Prime-v2 achieves 7.77 perplexity on OpenWebText, outperforming ARM (12.99), MDM (18.94), and MDM (13.41)
• Score: 26.967863200265494
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Masked diffusion models (MDM) exhibit superior generalization when learned using a Partial masking scheme (Prime). This approach converts tokens into sub-tokens and models the diffusion process at the sub-token level. We identify two limitations of the MDM-Prime framework. First, we lack tools to guide the hyperparameter choice of the token granularity in the subtokenizer. Second, we find that the function form of the subtokenizer significantly degrades likelihood estimation when paired with commonly used Byte-Pair-Encoding (BPE) tokenizers. To address these limitations, we study the tightness of the variational bound in MDM-Prime and develop MDM-Prime-v2, a masked diffusion language model which incorporates Binary Encoding and Index Shuffling. Our scaling analysis reveals that MDM-Prime-v2 is 21.8$\times$ more compute-efficient than autoregressive models (ARM). In compute-optimal comparisons, MDM-Prime-v2 achieves 7.77 perplexity on OpenWebText, outperforming ARM (12.99), MDM (18.94), and MDM-Prime (13.41). When extending the model size to 1.1B parameters, our model further demonstrates superior zero-shot accuracy on various commonsense reasoning tasks.

Related papers

• Understanding and Accelerating the Training of Masked Diffusion Language Models [104.60946175023008]
  Masked diffusion models (MDMs) have emerged as a promising alternative to autoregressive models (ARMs) for language modeling.<n>We show how to accelerate standard MDM training while maintaining its final performance.<n>We also show faster improvements in generative perplexity, zero-shot perplexity, and downstream task performance on various benchmarks.
  arXiv  Detail & Related papers  (2026-05-13T05:29:38Z)
• Infinite Mask Diffusion for Few-Step Distillation [16.340935269530842]
  Masked Diffusion Models (MDMs) have emerged as a promising alternative to autoregressive models in language modeling.<n>We propose the Infinite Mask Diffusion Model (IMDM) to mitigate the theoretical bound while inheriting the benefits of MDMs.<n>IMDM surpasses existing few-step distillation methods at small step counts on LM1B and OpenWebText.
  arXiv  Detail & Related papers  (2026-05-11T13:07:03Z)
• DMax: Aggressive Parallel Decoding for dLLMs [77.24184219948337]
  We present DMax, a new paradigm for efficient diffusion language models (dLLMs)<n>It mitigates error accumulation in parallel decoding, enabling aggressive decoding while preserving generation quality.<n>At the core of our approach is On-Policy Uniform Training, a novel training strategy that efficiently unifies masked and uniform dLLMs.
  arXiv  Detail & Related papers  (2026-04-09T14:35:42Z)
• Unifying Masked Diffusion Models with Various Generation Orders and Beyond [56.70289720766803]
  Masked diffusion models (MDMs) are a potential alternative to autoregressive models (ARMs) for language generation.<n>We propose order-expressive masked diffusion model (OeMDM) for a broad class of diffusion generative processes.<n>We introduce learnable-order masked diffusion model (LoMDM) which jointly learns the generation ordering and diffusion backbone.
  arXiv  Detail & Related papers  (2026-02-02T13:54:32Z)
• Soft-Masked Diffusion Language Models [35.191030145577145]
  We introduce soft-masking (SM), a novel method that dynamically blends the embedding of the mask token with the embeddings of the top-$k$ predicted tokens.<n>We demonstrate that continuing pretraining a 169M parameter model with SM leads to improved perplexity and MAUVE scores.<n>We finetune two state-of-the-art diffusion models, Dream-7B and Dream-Coder-7B, with SM.
  arXiv  Detail & Related papers  (2025-10-20T06:42:03Z)
• Sequential Diffusion Language Models [110.06562906987052]
  Diffusion language models (DLMs) have strong theoretical efficiency but are limited by fixed-length decoding and incompatibility with key-value caches.<n>We introduce Next Sequence Prediction (NSP), which unifies next-token and next-block prediction.<n>We propose Sequential Diffusion Language Model (SDLM), which can retrofit pre-trained autoregressive language models (ALMs) at minimal cost.
  arXiv  Detail & Related papers  (2025-09-28T17:59:15Z)
• Any-Order Flexible Length Masked Diffusion [53.89217188409148]
  Masked diffusion models (MDMs) have recently emerged as a promising alternative to autoregressive models over discrete domains.<n>We introduce Flexible Masked Diffusion Models (FlexMDMs), a discrete diffusion paradigm that simultaneously can model sequences of flexible length.<n>We show that FlexMDMs match MDMs in perplexity while modeling length statistics with much higher fidelity.
  arXiv  Detail & Related papers  (2025-08-31T23:34:53Z)
• Beyond Masked and Unmasked: Discrete Diffusion Models via Partial Masking [28.55159825491572]
  Masked diffusion models (MDM) are powerful generative models for discrete data that generate samples by progressively unmasking tokens in a sequence.<n>We propose the Partial masking scheme (Prime), which augments MDM by allowing tokens to take intermediate states between the masked and unmasked states.<n>Our method demonstrates superior performance across a diverse set of generative modeling tasks.
  arXiv  Detail & Related papers  (2025-05-24T04:16:40Z)
• Large Language Diffusion Models [93.26422905620008]
  Large language models (LLMs) are widely regarded as relying on autoregressive models (ARMs)<n>We introduce LLaDA, a diffusion model trained from scratch under the pre-training and supervised fine-tuning paradigm.<n>Across extensive benchmarks on general tasks, math, code, and so on, LLaDA demonstrates strong scalability and performs comparably to our self-constructed ARM baselines.
  arXiv  Detail & Related papers  (2025-02-14T08:23:51Z)
• Scaling up Masked Diffusion Models on Text [43.16800764711572]
  Masked diffusion models (MDMs) have shown promise in language modeling.<n>This paper establishes the first scaling law for MDMs.<n>We train a family of MDMs with up to 1.1 billion (B) parameters to evaluate their performance against larger sizes.
  arXiv  Detail & Related papers  (2024-10-24T08:01:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.