Lookahead Diffusion Probabilistic Models for Refining Mean Estimation
- URL: http://arxiv.org/abs/2304.11312v1
- Date: Sat, 22 Apr 2023 03:58:08 GMT
- Title: Lookahead Diffusion Probabilistic Models for Refining Mean Estimation
- Authors: Guoqiang Zhang, Niwa Kenta, W. Bastiaan Kleijn
- Abstract summary: We propose LA-DPMs to exploit the correlation in the outputs of the deep neural networks (DNNs) over subsequent timesteps in diffusion probabilistic models (DPMs)
A typical DPM first obtains an estimate of the original data sample $boldsymbolx$ by feeding the most recent state $boldsymbolz_i$ and index $i$ into the DNN model.
We propose to calculate a more accurate estimate for $boldsymbolx$ by performing extrapolation on the two estimates of $boldsymbolx$ that are obtained by feeding $
- Score: 23.424045204296437
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We propose lookahead diffusion probabilistic models (LA-DPMs) to exploit the
correlation in the outputs of the deep neural networks (DNNs) over subsequent
timesteps in diffusion probabilistic models (DPMs) to refine the mean
estimation of the conditional Gaussian distributions in the backward process. A
typical DPM first obtains an estimate of the original data sample
$\boldsymbol{x}$ by feeding the most recent state $\boldsymbol{z}_i$ and index
$i$ into the DNN model and then computes the mean vector of the conditional
Gaussian distribution for $\boldsymbol{z}_{i-1}$. We propose to calculate a
more accurate estimate for $\boldsymbol{x}$ by performing extrapolation on the
two estimates of $\boldsymbol{x}$ that are obtained by feeding
$(\boldsymbol{z}_{i+1},i+1)$ and $(\boldsymbol{z}_{i},i)$ into the DNN model.
The extrapolation can be easily integrated into the backward process of
existing DPMs by introducing an additional connection over two consecutive
timesteps, and fine-tuning is not required. Extensive experiments showed that
plugging in the additional connection into DDPM, DDIM, DEIS, S-PNDM, and
high-order DPM-Solvers leads to a significant performance gain in terms of FID
score.
Related papers
- Exploiting Pre-trained Models for Drug Target Affinity Prediction with Nearest Neighbors [58.661454334877256]
Drug-Target binding Affinity (DTA) prediction is essential for drug discovery.
Despite the application of deep learning methods to DTA prediction, the achieved accuracy remain suboptimal.
We propose $k$NN-DTA, a non-representation embedding-based retrieval method adopted on a pre-trained DTA prediction model.
arXiv Detail & Related papers (2024-07-21T15:49:05Z) - Convergence Analysis of Probability Flow ODE for Score-based Generative Models [5.939858158928473]
We study the convergence properties of deterministic samplers based on probability flow ODEs from both theoretical and numerical perspectives.
We prove the total variation between the target and the generated data distributions can be bounded above by $mathcalO(d3/4delta1/2)$ in the continuous time level.
arXiv Detail & Related papers (2024-04-15T12:29:28Z) - Accelerating Convergence of Score-Based Diffusion Models, Provably [44.11766377798812]
Score-based diffusion models often suffer from low sampling speed due to extensive function evaluations needed during the sampling phase.
We design novel training-free algorithms to accelerate popular deterministic (i.e., DDIM) and (i.e., DDPM) samplers.
Our theory accommodates $ell$-accurate score estimates, and does not require log-concavity or smoothness on the target distribution.
arXiv Detail & Related papers (2024-03-06T17:02:39Z) - SPD-DDPM: Denoising Diffusion Probabilistic Models in the Symmetric
Positive Definite Space [47.65912121120524]
We propose a novel generative model, termed SPD-DDPM, to handle large-scale data.
Our model is able to estimate $p(X)$ unconditionally and flexibly without giving $y$.
Experiment results on toy data and real taxi data demonstrate that our models effectively fit the data distribution both unconditionally and unconditionally.
arXiv Detail & Related papers (2023-12-13T15:08:54Z) - Nearly $d$-Linear Convergence Bounds for Diffusion Models via Stochastic
Localization [40.808942894229325]
We provide the first convergence bounds which are linear in the data dimension.
We show that diffusion models require at most $tilde O(fracd log2(1/delta)varepsilon2)$ steps to approximate an arbitrary distribution.
arXiv Detail & Related papers (2023-08-07T16:01:14Z) - Towards Faster Non-Asymptotic Convergence for Diffusion-Based Generative
Models [49.81937966106691]
We develop a suite of non-asymptotic theory towards understanding the data generation process of diffusion models.
In contrast to prior works, our theory is developed based on an elementary yet versatile non-asymptotic approach.
arXiv Detail & Related papers (2023-06-15T16:30:08Z) - On Calibrating Diffusion Probabilistic Models [78.75538484265292]
diffusion probabilistic models (DPMs) have achieved promising results in diverse generative tasks.
We propose a simple way for calibrating an arbitrary pretrained DPM, with which the score matching loss can be reduced and the lower bounds of model likelihood can be increased.
Our calibration method is performed only once and the resulting models can be used repeatedly for sampling.
arXiv Detail & Related papers (2023-02-21T14:14:40Z) - CARD: Classification and Regression Diffusion Models [51.0421331214229]
We introduce classification and regression diffusion (CARD) models, which combine a conditional generative model and a pre-trained conditional mean estimator.
We demonstrate the outstanding ability of CARD in conditional distribution prediction with both toy examples and real-world datasets.
arXiv Detail & Related papers (2022-06-15T03:30:38Z) - $p$-Generalized Probit Regression and Scalable Maximum Likelihood
Estimation via Sketching and Coresets [74.37849422071206]
We study the $p$-generalized probit regression model, which is a generalized linear model for binary responses.
We show how the maximum likelihood estimator for $p$-generalized probit regression can be approximated efficiently up to a factor of $(1+varepsilon)$ on large data.
arXiv Detail & Related papers (2022-03-25T10:54:41Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.