Transformers Learn the Optimal DDPM Denoiser for Multi-Token GMMs

• URL: http://arxiv.org/abs/2604.10074v1
• Date: Sat, 11 Apr 2026 07:46:15 GMT
• Title: Transformers Learn the Optimal DDPM Denoiser for Multi-Token GMMs
• Abstract summary: This paper provides the first convergence analysis for training transformer-based diffusion models.<n>We consider the population Denoising Diffusion Probabilistic Model (DDPM) objective for denoising data.<n>A deeper investigation reveals that trained a mean denoising module enables the trained model to approximate the oracle Minimum Mean Squared Error (MMSE) of the injected noise in the diffusion steps.
• Score: 13.741630476895773
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Transformer-based diffusion models have demonstrated remarkable performance at generating high-quality samples. However, our theoretical understanding of the reasons for this success remains limited. For instance, existing models are typically trained by minimizing a denoising objective, which is equivalent to fitting the score function of the training data. However, we do not know why transformer-based models can match the score function for denoising, or why gradient-based methods converge to the optimal denoising model despite the non-convex loss landscape. To the best of our knowledge, this paper provides the first convergence analysis for training transformer-based diffusion models. More specifically, we consider the population Denoising Diffusion Probabilistic Model (DDPM) objective for denoising data that follow a multi-token Gaussian mixture distribution. We theoretically quantify the required number of tokens per data point and training iterations for the global convergence towards the Bayes optimal risk of the denoising objective, thereby achieving a desired score matching error. A deeper investigation reveals that the self-attention module of the trained transformer implements a mean denoising mechanism that enables the trained model to approximate the oracle Minimum Mean Squared Error (MMSE) estimator of the injected noise in the diffusion steps. Numerical experiments validate these findings.

Related papers

• ASSERT: Adaptive Stochastic Sampling for Robust Diffusion Models on Analog Compute-in-Memory Hardware [13.123394042691642]
  Diffusion models achieve strong image generation quality but incur high iterative denoising costs.<n>We show that the early, high-noise denoising stage is substantially more vulnerable than the final refinement stage.<n>We propose ASSERT, a training sampler that uses higher deterministicity early and smoothly denoising.
  arXiv  Detail & Related papers  (2026-09-01T09:13:10Z)
• From Atoms to Entropy: Optimal Noise Allocation for Diffusion Training in the Convex Regime [52.459513910242485]
  We develop a general statistical framework for studying optimal noise-level allocation in diffusion training.<n>We show that the optimized training schedule admits an atomic minimizer, concentrated on finitely many noise levels.<n>We then evaluate the entropic schedule in larger-scale experiments, where full schedule optimization is currently intractable.
  arXiv  Detail & Related papers  (2026-07-10T20:59:06Z)
• NoiseRater: Meta-Learned Noise Valuation for Diffusion Model Training [81.58321000203712]
  NoiseRater is a meta-learning framework for instance-level noise valuation in diffusion model training.<n>NoiseRater assigns importance scores to individual noise realizations conditioned on data and timestep.<n>Experiments on FFHQ and ImageNet demonstrate that not all noise samples contribute equally.
  arXiv  Detail & Related papers  (2026-05-02T19:43:16Z)
• Diffusion Models Generalize but Not in the Way You Might Think [0.42970700836450487]
  We show that although sufficiently large denoiser models show increasing memorization of the training set with increasing training time, the resulting denoising trajectories do not follow this trend.<n>We make use of a 2D toy diffusion model to show that overfitting at intermediate noise levels is largely determined by model error and the density of the data support.
  arXiv  Detail & Related papers  (2026-03-12T21:02:17Z)
• Inference-Time Scaling of Diffusion Language Models with Particle Gibbs Sampling [70.8832906871441]
  We study how to steer generation toward desired rewards without retraining the models.<n>Prior methods typically resample or filter within a single denoising trajectory, optimizing rewards step-by-step without trajectory-level refinement.<n>We introduce particle Gibbs sampling for diffusion language models (PG-DLM), a novel inference-time algorithm enabling trajectory-level refinement while preserving generation perplexity.
  arXiv  Detail & Related papers  (2025-07-11T08:00:47Z)
• Noise Conditional Variational Score Distillation [60.38982038894823]
  Noise Conditional Variational Score Distillation (NCVSD) is a novel method for distilling pretrained diffusion models into generative denoisers.<n>By integrating this insight into the Variational Score Distillation framework, we enable scalable learning of generative denoisers.
  arXiv  Detail & Related papers  (2025-06-11T06:01:39Z)
• Dimension-free Score Matching and Time Bootstrapping for Diffusion Models [19.62665684173391]
  Diffusion models generate samples by estimating the score function of the target distribution at various noise levels.<n>We introduce a martingale-based error decomposition and sharp variance bounds, enabling efficient learning from dependent data.<n>Building on these insights, we propose Bootstrapped Score Matching (BSM), a variance reduction technique that leverages previously learned scores to improve accuracy at higher noise levels.
  arXiv  Detail & Related papers  (2025-02-14T18:32:22Z)
• Beyond Fixed Horizons: A Theoretical Framework for Adaptive Denoising Diffusions [1.9116784879310031]
  We introduce a new class of generative diffusion models that achieve a time-homogeneous structure for both the noising and denoising processes.<n>A key feature of the model is its adaptability to the target data, enabling a variety of downstream tasks using a pre-trained unconditional generative model.
  arXiv  Detail & Related papers  (2025-01-31T18:23:27Z)
• Impact of Noisy Supervision in Foundation Model Learning [91.56591923244943]
  This paper is the first work to comprehensively understand and analyze the nature of noise in pre-training datasets.<n>We propose a tuning method (NMTune) to affine the feature space to mitigate the malignant effect of noise and improve generalization.
  arXiv  Detail & Related papers  (2024-03-11T16:22:41Z)
• Interpreting and Improving Diffusion Models from an Optimization Perspective [4.5993996573872185]
  We use this observation to interpret denoising diffusion models as approximate gradient descent applied to the Euclidean distance function. We propose a new gradient-estimation sampler, generalizing DDIM using insights from our theoretical results.
  arXiv  Detail & Related papers  (2023-06-08T00:56:33Z)
• Denoising Distantly Supervised Named Entity Recognition via a Hypergeometric Probabilistic Model [26.76830553508229]
  Hypergeometric Learning (HGL) is a denoising algorithm for distantly supervised named entity recognition. HGL takes both noise distribution and instance-level confidence into consideration. Experiments show that HGL can effectively denoise the weakly-labeled data retrieved from distant supervision.
  arXiv  Detail & Related papers  (2021-06-17T04:01:25Z)
• Generative Modeling with Denoising Auto-Encoders and Langevin Sampling [88.83704353627554]
  We show that both DAE and DSM provide estimates of the score of the smoothed population density. We then apply our results to the homotopy method of arXiv:1907.05600 and provide theoretical justification for its empirical success.
  arXiv  Detail & Related papers  (2020-01-31T23:50:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.