Remasking Discrete Diffusion Models with Inference-Time Scaling

• URL: http://arxiv.org/abs/2503.00307v1
• Date: Sat, 01 Mar 2025 02:37:51 GMT
• Title: Remasking Discrete Diffusion Models with Inference-Time Scaling
• Authors: Guanghan Wang, Yair Schiff, Subham Sekhar Sahoo, Volodymyr Kuleshov,
• Abstract summary: We introduce the remasking diffusion model (ReMDM) sampler, a method that can be applied to pretrained masked diffusion models in a principled way.<n>Most interestingly, ReMDM endows discrete diffusion with a form of inference-time compute scaling.
• Score: 12.593164604625384
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Part of the success of diffusion models stems from their ability to perform iterative refinement, i.e., repeatedly correcting outputs during generation. However, modern masked discrete diffusion lacks this capability: when a token is generated, it cannot be updated again, even when it introduces an error. Here, we address this limitation by introducing the remasking diffusion model (ReMDM) sampler, a method that can be applied to pretrained masked diffusion models in a principled way and that is derived from a discrete diffusion model with a custom remasking backward process. Most interestingly, ReMDM endows discrete diffusion with a form of inference-time compute scaling. By increasing the number of sampling steps, ReMDM generates natural language outputs that approach the quality of autoregressive models, whereas when the computation budget is limited, ReMDM better maintains quality. ReMDM also improves sample quality of masked diffusion models for discretized images, and in scientific domains such as molecule design, ReMDM facilitates diffusion guidance and pushes the Pareto frontier of controllability relative to classical masking and uniform noise diffusion. We provide the code along with a blog post on the project page: https://remdm.github.io.

Related papers

• Generalized Interpolating Discrete Diffusion [65.74168524007484]
  Masked diffusion is a popular choice due to its simplicity and effectiveness. We derive the theoretical backbone of a family of general interpolating discrete diffusion processes. Exploiting GIDD's flexibility, we explore a hybrid approach combining masking and uniform noise.
  arXiv  Detail & Related papers  (2025-03-06T14:30:55Z)
• One-for-More: Continual Diffusion Model for Anomaly Detection [61.12622458367425]
  Anomaly detection methods utilize diffusion models to generate or reconstruct normal samples when given arbitrary anomaly images. Our study found that the diffusion model suffers from severe faithfulness hallucination'' and catastrophic forgetting'' We propose a continual diffusion model that uses gradient projection to achieve stable continual learning.
  arXiv  Detail & Related papers  (2025-02-27T07:47:27Z)
• RDPM: Solve Diffusion Probabilistic Models via Recurrent Token Prediction [17.005198258689035]
  Diffusion Probabilistic Models (DPMs) have emerged as the de facto approach for high-fidelity image synthesis. We introduce a novel generative framework, the Recurrent Diffusion Probabilistic Model (RDPM), which enhances the diffusion process through a recurrent token prediction mechanism.
  arXiv  Detail & Related papers  (2024-12-24T12:28:19Z)
• ACDiT: Interpolating Autoregressive Conditional Modeling and Diffusion Transformer [95.80384464922147]
  Continuous visual generation requires the full-sequence diffusion-based approach.<n>We present ACDiT, an Autoregressive blockwise Conditional Diffusion Transformer.<n>We demonstrate that ACDiT can be seamlessly used in visual understanding tasks despite being trained on the diffusion objective.
  arXiv  Detail & Related papers  (2024-12-10T18:13:20Z)
• Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling [47.82616476928464]
  Masked diffusion models (MDMs) have emerged as a popular research topic for generative modeling of discrete data.<n>We show that both training and sampling of MDMs are theoretically free from the time variable.<n>We identify, for the first time, an underlying numerical issue, even with the commonly used 32-bit floating-point precision.
  arXiv  Detail & Related papers  (2024-09-04T17:48:19Z)
• Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-Based Decoding [84.3224556294803]
  Diffusion models excel at capturing the natural design spaces of images, molecules, DNA, RNA, and protein sequences. We aim to optimize downstream reward functions while preserving the naturalness of these design spaces. Our algorithm integrates soft value functions, which looks ahead to how intermediate noisy states lead to high rewards in the future.
  arXiv  Detail & Related papers  (2024-08-15T16:47:59Z)
• SparseDM: Toward Sparse Efficient Diffusion Models [20.783533300147866]
  We propose a method based on the improved Straight-Through Estimator to improve the deployment efficiency of diffusion models. Experimental results on a Transformer and UNet-based diffusion models demonstrate that our method reduces MACs by 50% while maintaining FID.
  arXiv  Detail & Related papers  (2024-04-16T10:31:06Z)
• UDPM: Upsampling Diffusion Probabilistic Models [33.51145642279836]
  Denoising Diffusion Probabilistic Models (DDPM) have recently gained significant attention. DDPMs generate high-quality samples from complex data distributions by defining an inverse process. Unlike generative adversarial networks (GANs), the latent space of diffusion models is less interpretable. In this work, we propose to generalize the denoising diffusion process into an Upsampling Diffusion Probabilistic Model (UDPM)
  arXiv  Detail & Related papers  (2023-05-25T17:25:14Z)
• Diffusion Models as Masked Autoencoders [52.442717717898056]
  We revisit generatively pre-training visual representations in light of recent interest in denoising diffusion models. While directly pre-training with diffusion models does not produce strong representations, we condition diffusion models on masked input and formulate diffusion models as masked autoencoders (DiffMAE) We perform a comprehensive study on the pros and cons of design choices and build connections between diffusion models and masked autoencoders.
  arXiv  Detail & Related papers  (2023-04-06T17:59:56Z)
• Cold Diffusion: Inverting Arbitrary Image Transforms Without Noise [52.59444045853966]
  We show that an entire family of generative models can be constructed by varying the choice of image degradation. The success of fully deterministic models calls into question the community's understanding of diffusion models.
  arXiv  Detail & Related papers  (2022-08-19T15:18:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.