Uniform Diffusion Models Revisited: Leave-One-Out Denoiser and Absorbing State Reformulation

• URL: http://arxiv.org/abs/2605.22765v1
• Date: Thu, 21 May 2026 17:27:19 GMT
• Title: Uniform Diffusion Models Revisited: Leave-One-Out Denoiser and Absorbing State Reformulation
• Abstract summary: We show that the standard plug-in bridge parameterization for UDM is not optimized by the denoising posterior.<n>We also introduce an absorbing-state reformulation of uniform diffusion that preserves the UDM joint law while decomposing it into masked-diffusion-like sampling operations.
• Score: 72.05592785529404
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Discrete diffusion models are often trained through clean-data prediction, but the prediction can be used in different ways to define the reverse dynamics. In Masked Diffusion Models (MDM) these choices largely coincide, whereas in Uniform Diffusion Models (UDM) they do not. We show that the standard plug-in bridge parameterization for UDM is not optimized by the denoising posterior, but by a leave-one-out posterior that predicts each clean token without using its own noisy observation. This identifies a mismatch between the plug-in ELBO and the usual cross-entropy denoising objective. We characterize the leave-one-out target and derive exact conversions between the denoiser, the leave-one-out posterior, and the score. These conversions allow us to disentangle parameterization and training objective. Our results also lead to inference improvements without any additional training through an informed predictor-corrector sampler and improved temperature sampling based on the leave-one-out predictor. We further introduce an absorbing-state reformulation of uniform diffusion that preserves the UDM joint law while decomposing it into masked-diffusion-like sampling operations, with simpler denoising posteriors, carry-over unmasking, and a natural remasking mechanism. On language modeling, leave-one-out parameterizations consistently improve UDM generation, while the absorbing construction matches or surpasses masked diffusion. These results suggest that the empirical gap between masked and uniform diffusion is driven less by the choice of marginals themselves than by parameterization and sampling design. The code and models can be found at https://github.com/samsongourevitch/rev_udm.

Related papers

• Impute-EM: Native Mixed-State Diffusion Models for Heterogeneous Data Imputation [71.74203645336385]
  Impute-EM alternates between imputing missing entries with the current model and refitting a diffusion backbone on completed data.<n>We characterize the update and show that the observed mask-indexed marginals match the targets at the limit.
  arXiv  Detail & Related papers  (2026-09-14T09:39:13Z)
• Simple Self-Conditioning Adaptation for Masked Diffusion Models [45.533070677326805]
  Masked diffusion models (MDMs) generate discrete sequences by iterative denoising under an absorbing masking process.<n>This paper proposes a simple, yet effective, post-training adaptation for MDMs that conditions each denoising step on the model's own previous clean-state predictions.<n>The resulting method, called Self-Conditioned Masked Diffusion Models (SCMDM), requires minimal architectural change.
  arXiv  Detail & Related papers  (2026-04-28T19:34:04Z)
• Interpolating Discrete Diffusion Models with Controllable Resampling [49.771004594828405]
  Masked diffusion models suffer from irreversible errors due to early unmasking.<n>Uniform diffusion models, despite enabling self-correction, often yield low-quality samples.<n>We introduce Interpolating Discrete Diffusion Model, that improves diffusion by reducing dependence on intermediate latent states.
  arXiv  Detail & Related papers  (2026-04-19T07:55:57Z)
• Continuously Augmented Discrete Diffusion model for Categorical Generative Modeling [87.34677262370924]
  Standard discrete diffusion models treat all unobserved states identically by mapping them to an absorbing [MASK] token.<n>This creates an 'information void' where semantic information that could be inferred from unmasked tokens is lost between denoising steps.<n>We introduce Continuously Augmented Discrete Diffusion, a framework that augments the discrete state space with a paired diffusion in a continuous latent space.
  arXiv  Detail & Related papers  (2025-10-01T18:00:56Z)
• Generalized Interpolating Discrete Diffusion [65.74168524007484]
  Masked diffusion is a popular choice due to its simplicity and effectiveness.<n>We generalize a new family of general interpolating discrete diffusion (GIDD) which offers greater flexibility in the design of the noising processes.<n>Exploiting GIDD's flexibility, we explore a hybrid approach combining masking and uniform noise, leading to improved sample quality.
  arXiv  Detail & Related papers  (2025-03-06T14:30:55Z)
• Remasking Discrete Diffusion Models with Inference-Time Scaling [21.362017006523086]
  We introduce the remasking diffusion model (ReMDM) sampler, a method that can be applied to pretrained masked diffusion models in a principled way.<n>Most interestingly, ReMDM endows discrete diffusion with a form of inference-time compute scaling.
  arXiv  Detail & Related papers  (2025-03-01T02:37:51Z)
• Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling [47.82616476928464]
  Masked diffusion models (MDMs) have emerged as a popular research topic for generative modeling of discrete data.<n>We show that both training and sampling of MDMs are theoretically free from the time variable.<n>We identify, for the first time, an underlying numerical issue, even with the commonly used 32-bit floating-point precision.
  arXiv  Detail & Related papers  (2024-09-04T17:48:19Z)
• Denoising Diffusion Variational Inference: Diffusion Models as Expressive Variational Posteriors [11.01598521921903]
  DDVI is a black-box variational inference algorithm for latent variable models.<n>We introduce an expressive class of diffusion-based variational posteriors.<n>We train these posteriors with a novel regularized evidence lower bound.
  arXiv  Detail & Related papers  (2024-01-05T10:27:44Z)
• Unscented Autoencoder [3.0108936184913295]
  Variational Autoencoder (VAE) is a seminal approach in deep generative modeling with latent variables. We apply the Unscented Transform (UT) -- a well-known distribution approximation used in the Unscented Kalman Filter (UKF) from the field of filtering. We derive a novel, deterministic-sampling flavor of the VAE, the Unscented Autoencoder (UAE), trained purely with regularization-like terms on the per-sample posterior.
  arXiv  Detail & Related papers  (2023-06-08T14:53:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.