Di$\mathtt{[M]}$O: Distilling Masked Diffusion Models into One-step Generator

• URL: http://arxiv.org/abs/2503.15457v1
• Date: Wed, 19 Mar 2025 17:36:54 GMT
• Title: Di$\mathtt{[M]}$O: Distilling Masked Diffusion Models into One-step Generator
• Authors: Yuanzhi Zhu, Xi Wang, Stéphane Lathuilière, Vicky Kalogeiton,
• Abstract summary: Masked Diffusion Models (MDMs) have emerged as a powerful generative modeling technique.<n>We propose Di$mathtt[M]$O, a novel approach that distills masked diffusion models into a one-step generator.<n>We show Di$mathtt[M]$O's effectiveness on both class-conditional and text-conditional image generation.
• Score: 22.88494918435088
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Masked Diffusion Models (MDMs) have emerged as a powerful generative modeling technique. Despite their remarkable results, they typically suffer from slow inference with several steps. In this paper, we propose Di$\mathtt{[M]}$O, a novel approach that distills masked diffusion models into a one-step generator. Di$\mathtt{[M]}$O addresses two key challenges: (1) the intractability of using intermediate-step information for one-step generation, which we solve through token-level distribution matching that optimizes model output logits by an 'on-policy framework' with the help of an auxiliary model; and (2) the lack of entropy in the initial distribution, which we address through a token initialization strategy that injects randomness while maintaining similarity to teacher training distribution. We show Di$\mathtt{[M]}$O's effectiveness on both class-conditional and text-conditional image generation, impressively achieving performance competitive to multi-step teacher outputs while drastically reducing inference time. To our knowledge, we are the first to successfully achieve one-step distillation of masked diffusion models and the first to apply discrete distillation to text-to-image generation, opening new paths for efficient generative modeling.

Related papers

• Transition Matching Distillation for Fast Video Generation [63.1049790376783]
  We present Transition Matching Distillation (TMD), a novel framework for distilling video diffusion models into efficient few-step generators.<n>TMD matches the multi-step denoising trajectory of a diffusion model with a few-step probability transition process.<n>TMD provides a flexible and strong trade-off between generation speed and visual quality.
  arXiv  Detail & Related papers  (2026-01-14T21:30:03Z)
• Bridging the Discrete-Continuous Gap: Unified Multimodal Generation via Coupled Manifold Discrete Absorbing Diffusion [60.186310080523135]
  Bifurcation of generative modeling into autoregressive approaches for discrete data (text) and diffusion approaches for continuous data (images) hinders development of truly unified multimodal systems.<n>We propose textbfCoM-DAD, a novel probabilistic framework that reformulates multimodal generation as a hierarchical dual-process.<n>Our method demonstrates superior stability over standard masked modeling, establishing a new paradigm for scalable, unified text-image generation.
  arXiv  Detail & Related papers  (2026-01-07T16:21:19Z)
• Soft-Di[M]O: Improving One-Step Discrete Image Generation with Soft Embeddings [35.979608265594685]
  One-step generators distilled from Masked Diffusion Models (MDMs) compress multiple sampling steps into a single forward pass.<n>They inherit modeling bias from the teacher, and their discrete token outputs block gradient flow.<n>We introduce soft embeddings, a simple relaxation that replaces discrete tokens with the expected embeddings under the generator's output distribution.
  arXiv  Detail & Related papers  (2025-09-26T20:51:20Z)
• The Diffusion Duality [11.823724329261053]
  Uniform-state diffusion processes naturally emerge from an underlying Gaussian diffusion.<n>Models trained with curriculum learning surpass autoregressive models in zero-shot perplexity on 3 of 7 benchmarks.<n>We present Discrete Consistency Distillation, which adapts consistency distillation from the continuous to the discrete setting.
  arXiv  Detail & Related papers  (2025-06-12T16:55:35Z)
• Unifying Autoregressive and Diffusion-Based Sequence Generation [2.3923884480793673]
  We present extensions to diffusion-based sequence generation models, blurring the line with autoregressive language models.<n>We introduce hyperschedules, which assign distinct noise schedules to individual token positions.<n>Second, we propose two hybrid token-wise noising processes that interpolate between absorbing and uniform processes, enabling the model to fix past mistakes.
  arXiv  Detail & Related papers  (2025-04-08T20:32:10Z)
• Feynman-Kac Correctors in Diffusion: Annealing, Guidance, and Product of Experts [64.34482582690927]
  We provide an efficient and principled method for sampling from a sequence of annealed, geometric-averaged, or product distributions derived from pretrained score-based models. We propose Sequential Monte Carlo (SMC) resampling algorithms that leverage inference-time scaling to improve sampling quality.
  arXiv  Detail & Related papers  (2025-03-04T17:46:51Z)
• Remasking Discrete Diffusion Models with Inference-Time Scaling [12.593164604625384]
  We introduce the remasking diffusion model (ReMDM) sampler, a method that can be applied to pretrained masked diffusion models in a principled way. Most interestingly, ReMDM endows discrete diffusion with a form of inference-time compute scaling.
  arXiv  Detail & Related papers  (2025-03-01T02:37:51Z)
• Fast Solvers for Discrete Diffusion Models: Theory and Applications of High-Order Algorithms [31.42317398879432]
  Current inference approaches mainly fall into two categories: exact simulation and approximate methods such as $tau$-leaping. In this work, we advance the latter category by tailoring the first extension of high-order numerical inference schemes to discrete diffusion models. We rigorously analyze the proposed schemes and establish the second-order accuracy of the $theta$-trapezoidal method in KL divergence.
  arXiv  Detail & Related papers  (2025-02-01T00:25:21Z)
• RDPM: Solve Diffusion Probabilistic Models via Recurrent Token Prediction [17.005198258689035]
  Diffusion Probabilistic Models (DPMs) have emerged as the de facto approach for high-fidelity image synthesis. We introduce a novel generative framework, the Recurrent Diffusion Probabilistic Model (RDPM), which enhances the diffusion process through a recurrent token prediction mechanism.
  arXiv  Detail & Related papers  (2024-12-24T12:28:19Z)
• One-Step Diffusion Distillation through Score Implicit Matching [74.91234358410281]
  We present Score Implicit Matching (SIM) a new approach to distilling pre-trained diffusion models into single-step generator models. SIM shows strong empirical performances for one-step generators. By applying SIM to a leading transformer-based diffusion model, we distill a single-step generator for text-to-image generation.
  arXiv  Detail & Related papers  (2024-10-22T08:17:20Z)
• Local Flow Matching Generative Models [19.859984725284896]
  Local Flow Matching is a computational framework for density estimation based on flow-based generative models. $textttLFM$ employs a simulation-free scheme and incrementally learns a sequence of Flow Matching sub-models. We demonstrate the improved training efficiency and competitive generative performance of $textttLFM$ compared to FM.
  arXiv  Detail & Related papers  (2024-10-03T14:53:10Z)
• Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-Based Decoding [84.3224556294803]
  Diffusion models excel at capturing the natural design spaces of images, molecules, DNA, RNA, and protein sequences. We aim to optimize downstream reward functions while preserving the naturalness of these design spaces. Our algorithm integrates soft value functions, which looks ahead to how intermediate noisy states lead to high rewards in the future.
  arXiv  Detail & Related papers  (2024-08-15T16:47:59Z)
• EM Distillation for One-step Diffusion Models [65.57766773137068]
  We propose a maximum likelihood-based approach that distills a diffusion model to a one-step generator model with minimal loss of quality. We develop a reparametrized sampling scheme and a noise cancellation technique that together stabilizes the distillation process.
  arXiv  Detail & Related papers  (2024-05-27T05:55:22Z)
• Towards Faster Non-Asymptotic Convergence for Diffusion-Based Generative Models [49.81937966106691]
  We develop a suite of non-asymptotic theory towards understanding the data generation process of diffusion models. In contrast to prior works, our theory is developed based on an elementary yet versatile non-asymptotic approach.
  arXiv  Detail & Related papers  (2023-06-15T16:30:08Z)
• Image Generation with Multimodal Priors using Denoising Diffusion Probabilistic Models [54.1843419649895]
  A major challenge in using generative models to accomplish this task is the lack of paired data containing all modalities and corresponding outputs. We propose a solution based on a denoising diffusion probabilistic synthesis models to generate images under multi-model priors.
  arXiv  Detail & Related papers  (2022-06-10T12:23:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.