How Much is Enough? A Study on Diffusion Times in Score-based Generative
Models
- URL: http://arxiv.org/abs/2206.05173v1
- Date: Fri, 10 Jun 2022 15:09:46 GMT
- Title: How Much is Enough? A Study on Diffusion Times in Score-based Generative
Models
- Authors: Giulio Franzese and Simone Rossi and Lixuan Yang and Alessandro
Finamore and Dario Rossi and Maurizio Filippone and Pietro Michiardi
- Abstract summary: Current best practice advocates for a large T to ensure that the forward dynamics brings the diffusion sufficiently close to a known and simple noise distribution.
We show how an auxiliary model can be used to bridge the gap between the ideal and the simulated forward dynamics, followed by a standard reverse diffusion process.
- Score: 76.76860707897413
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Score-based diffusion models are a class of generative models whose dynamics
is described by stochastic differential equations that map noise into data.
While recent works have started to lay down a theoretical foundation for these
models, an analytical understanding of the role of the diffusion time T is
still lacking. Current best practice advocates for a large T to ensure that the
forward dynamics brings the diffusion sufficiently close to a known and simple
noise distribution; however, a smaller value of T should be preferred for a
better approximation of the score-matching objective and higher computational
efficiency. Starting from a variational interpretation of diffusion models, in
this work we quantify this trade-off, and suggest a new method to improve
quality and efficiency of both training and sampling, by adopting smaller
diffusion times. Indeed, we show how an auxiliary model can be used to bridge
the gap between the ideal and the simulated forward dynamics, followed by a
standard reverse diffusion process. Empirical results support our analysis; for
image data, our method is competitive w.r.t. the state-of-the-art, according to
standard sample quality metrics and log-likelihood.
Related papers
- Provable Statistical Rates for Consistency Diffusion Models [87.28777947976573]
Despite the state-of-the-art performance, diffusion models are known for their slow sample generation due to the extensive number of steps involved.
This paper contributes towards the first statistical theory for consistency models, formulating their training as a distribution discrepancy minimization problem.
arXiv Detail & Related papers (2024-06-23T20:34:18Z) - Data Attribution for Diffusion Models: Timestep-induced Bias in
Influence Estimation [58.20016784231991]
Diffusion models operate over a sequence of timesteps instead of instantaneous input-output relationships in previous contexts.
We present Diffusion-TracIn that incorporates this temporal dynamics and observe that samples' loss gradient norms are highly dependent on timestep.
We introduce Diffusion-ReTrac as a re-normalized adaptation that enables the retrieval of training samples more targeted to the test sample of interest.
arXiv Detail & Related papers (2024-01-17T07:58:18Z) - Debias the Training of Diffusion Models [53.49637348771626]
We provide theoretical evidence that the prevailing practice of using a constant loss weight strategy in diffusion models leads to biased estimation during the training phase.
We propose an elegant and effective weighting strategy grounded in the theoretically unbiased principle.
These analyses are expected to advance our understanding and demystify the inner workings of diffusion models.
arXiv Detail & Related papers (2023-10-12T16:04:41Z) - Soft Mixture Denoising: Beyond the Expressive Bottleneck of Diffusion
Models [76.46246743508651]
We show that current diffusion models actually have an expressive bottleneck in backward denoising.
We introduce soft mixture denoising (SMD), an expressive and efficient model for backward denoising.
arXiv Detail & Related papers (2023-09-25T12:03:32Z) - Diffusion Models are Minimax Optimal Distribution Estimators [49.47503258639454]
We provide the first rigorous analysis on approximation and generalization abilities of diffusion modeling.
We show that when the true density function belongs to the Besov space and the empirical score matching loss is properly minimized, the generated data distribution achieves the nearly minimax optimal estimation rates.
arXiv Detail & Related papers (2023-03-03T11:31:55Z) - Information-Theoretic Diffusion [18.356162596599436]
Denoising diffusion models have spurred significant gains in density modeling and image generation.
We introduce a new mathematical foundation for diffusion models inspired by classic results in information theory.
arXiv Detail & Related papers (2023-02-07T23:03:07Z) - Non-Uniform Diffusion Models [0.8602553195689513]
We show that non-uniform diffusion leads to multi-scale diffusion models which have similar structure to this of multi-scale normalizing flows.
We experimentally find that in the same or less training time, the multi-scale diffusion model achieves better FID score than the standard uniform diffusion model.
We also show that non-uniform diffusion leads to a novel estimator for the conditional score function which achieves on par performance with the state-of-the-art conditional denoising estimator.
arXiv Detail & Related papers (2022-07-20T09:59:28Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.