Evaluating the design space of diffusion-based generative models

• URL: http://arxiv.org/abs/2406.12839v4
• Date: Sun, 27 Oct 2024 21:51:56 GMT
• Title: Evaluating the design space of diffusion-based generative models
• Authors: Yuqing Wang, Ye He, Molei Tao,
• Abstract summary: This article provides a first quantitative understanding of the whole generation process. It conducts a non-asymptotic convergence analysis of denoising score matching under gradient descent. A refined sampling error analysis for variance exploding models is also provided.
• Score: 21.483299796597404
• License:
• Abstract: Most existing theoretical investigations of the accuracy of diffusion models, albeit significant, assume the score function has been approximated to a certain accuracy, and then use this a priori bound to control the error of generation. This article instead provides a first quantitative understanding of the whole generation process, i.e., both training and sampling. More precisely, it conducts a non-asymptotic convergence analysis of denoising score matching under gradient descent. In addition, a refined sampling error analysis for variance exploding models is also provided. The combination of these two results yields a full error analysis, which elucidates (again, but this time theoretically) how to design the training and sampling processes for effective generation. For instance, our theory implies a preference toward noise distribution and loss weighting in training that qualitatively agree with the ones used in [Karras et al., 2022]. It also provides perspectives on the choices of time and variance schedules in sampling: when the score is well trained, the design in [Song et al., 2021] is more preferable, but when it is less trained, the design in [Karras et al., 2022] becomes more preferable.

Related papers

• Theory on Score-Mismatched Diffusion Models and Zero-Shot Conditional Samplers [49.97755400231656]
  We present the first performance guarantee with explicit dimensional general score-mismatched diffusion samplers. We show that score mismatches result in an distributional bias between the target and sampling distributions, proportional to the accumulated mismatch between the target and training distributions. This result can be directly applied to zero-shot conditional samplers for any conditional model, irrespective of measurement noise.
  arXiv  Detail & Related papers  (2024-10-17T16:42:12Z)
• Informed Correctors for Discrete Diffusion Models [32.87362154118195]
  We propose a family of informed correctors that more reliably counteracts discretization error by leveraging information learned by the model. We also propose $k$-Gillespie's, a sampling algorithm that better utilizes each model evaluation, while still enjoying the speed and flexibility of $tau$-leaping. Across several real and synthetic datasets, we show that $k$-Gillespie's with informed correctors reliably produces higher quality samples at lower computational cost.
  arXiv  Detail & Related papers  (2024-07-30T23:29:29Z)
• Leveraging Uncertainty Estimates To Improve Classifier Performance [4.4951754159063295]
  Binary classification involves predicting the label of an instance based on whether the model score for the positive class exceeds a threshold chosen based on the application requirements. However, model scores are often not aligned with the true positivity rate. This is especially true when the training involves a differential sampling across classes or there is distributional drift between train and test settings.
  arXiv  Detail & Related papers  (2023-11-20T12:40:25Z)
• How Much is Enough? A Study on Diffusion Times in Score-based Generative Models [76.76860707897413]
  Current best practice advocates for a large T to ensure that the forward dynamics brings the diffusion sufficiently close to a known and simple noise distribution. We show how an auxiliary model can be used to bridge the gap between the ideal and the simulated forward dynamics, followed by a standard reverse diffusion process.
  arXiv  Detail & Related papers  (2022-06-10T15:09:46Z)
• On the Benefits of Large Learning Rates for Kernel Methods [110.03020563291788]
  We show that a phenomenon can be precisely characterized in the context of kernel methods. We consider the minimization of a quadratic objective in a separable Hilbert space, and show that with early stopping, the choice of learning rate influences the spectral decomposition of the obtained solution.
  arXiv  Detail & Related papers  (2022-02-28T13:01:04Z)
• Benign-Overfitting in Conditional Average Treatment Effect Prediction with Linear Regression [14.493176427999028]
  We study the benign overfitting theory in the prediction of the conditional average treatment effect (CATE) with linear regression models. We show that the T-learner fails to achieve the consistency except the random assignment, while the IPW-learner converges the risk to zero if the propensity score is known.
  arXiv  Detail & Related papers  (2022-02-10T18:51:52Z)
• A Variational Perspective on Diffusion-Based Generative Models and Score Matching [8.93483643820767]
  We derive a variational framework for likelihood estimation for continuous-time generative diffusion. We show that minimizing the score-matching loss is equivalent to maximizing a lower bound of the likelihood of the plug-in reverse SDE.
  arXiv  Detail & Related papers  (2021-06-05T05:50:36Z)
• Optimization Variance: Exploring Generalization Properties of DNNs [83.78477167211315]
  The test error of a deep neural network (DNN) often demonstrates double descent. We propose a novel metric, optimization variance (OV), to measure the diversity of model updates.
  arXiv  Detail & Related papers  (2021-06-03T09:34:17Z)
• On the Inherent Regularization Effects of Noise Injection During Training [12.614901374282868]
  We present a theoretical study of one particular way of random perturbation, which corresponds to injecting artificial noise to the training data. We provide a precise characterization of the training and generalization errors of such randomly perturbed learning problems on a random feature model.
  arXiv  Detail & Related papers  (2021-02-15T07:43:18Z)
• Goal-directed Generation of Discrete Structures with Conditional Generative Models [85.51463588099556]
  We introduce a novel approach to directly optimize a reinforcement learning objective, maximizing an expected reward. We test our methodology on two tasks: generating molecules with user-defined properties and identifying short python expressions which evaluate to a given target value.
  arXiv  Detail & Related papers  (2020-10-05T20:03:13Z)
• Good Classifiers are Abundant in the Interpolating Regime [64.72044662855612]
  We develop a methodology to compute precisely the full distribution of test errors among interpolating classifiers. We find that test errors tend to concentrate around a small typical value $varepsilon*$, which deviates substantially from the test error of worst-case interpolating model. Our results show that the usual style of analysis in statistical learning theory may not be fine-grained enough to capture the good generalization performance observed in practice.
  arXiv  Detail & Related papers  (2020-06-22T21:12:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.