Estimating the Optimal Covariance with Imperfect Mean in Diffusion Probabilistic Models

• URL: http://arxiv.org/abs/2206.07309v1
• Date: Wed, 15 Jun 2022 05:42:48 GMT
• Title: Estimating the Optimal Covariance with Imperfect Mean in Diffusion Probabilistic Models
• Authors: Fan Bao, Chongxuan Li, Jiacheng Sun, Jun Zhu, Bo Zhang
• Abstract summary: Diffusion probabilistic models (DPMs) are a class of powerful deep generative models (DGMs) Despite their success, the iterative generation process over the full timesteps is much less efficient than other DGMs such as GANs. We consider diagonal and full covariances to improve the expressive power of DPMs.
• Score: 37.18522296366212
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Diffusion probabilistic models (DPMs) are a class of powerful deep generative models (DGMs). Despite their success, the iterative generation process over the full timesteps is much less efficient than other DGMs such as GANs. Thus, the generation performance on a subset of timesteps is crucial, which is greatly influenced by the covariance design in DPMs. In this work, we consider diagonal and full covariances to improve the expressive power of DPMs. We derive the optimal result for such covariances, and then correct it when the mean of DPMs is imperfect. Both the optimal and the corrected ones can be decomposed into terms of conditional expectations over functions of noise. Building upon it, we propose to estimate the optimal covariance and its correction given imperfect mean by learning these conditional expectations. Our method can be applied to DPMs with both discrete and continuous timesteps. We consider the diagonal covariance in our implementation for computational efficiency. For an efficient practical implementation, we adopt a parameter sharing scheme and a two-stage training process. Empirically, our method outperforms a wide variety of covariance design on likelihood results, and improves the sample quality especially on a small number of timesteps.

Related papers

• Denoising diffusion probabilistic models are optimally adaptive to unknown low dimensionality [21.10158431913811]
  We investigate how the DDPM can achieve sampling speed-ups through automatic exploitation of intrinsic low dimensionality of data. We prove that the iteration complexity of the DDPM scales nearly linearly with $k$, which is optimal when using KL divergence to measure distributional discrepancy.
  arXiv  Detail & Related papers  (2024-10-24T14:36:12Z)
• Contractive Diffusion Probabilistic Models [5.217870815854702]
  Diffusion probabilistic models (DPMs) have emerged as a promising technique in generative modeling. We propose a new criterion -- the contraction property of backward sampling in the design of DPMs, leading to a novel class of contractive DPMs (CDPMs) We show that CDPM can leverage weights of pretrained DPMs by a simple transformation, and does not need retraining.
  arXiv  Detail & Related papers  (2024-01-23T21:51:51Z)
• DPM-OT: A New Diffusion Probabilistic Model Based on Optimal Transport [26.713392774427653]
  DPM-OT is a unified learning framework for fast DPMs with a direct expressway represented by OT map. It can generate high-quality samples within around 10 function evaluations. Experiments validate the effectiveness and advantages of DPM-OT in terms of speed and quality.
  arXiv  Detail & Related papers  (2023-07-21T02:28:54Z)
• On Calibrating Diffusion Probabilistic Models [78.75538484265292]
  diffusion probabilistic models (DPMs) have achieved promising results in diverse generative tasks. We propose a simple way for calibrating an arbitrary pretrained DPM, with which the score matching loss can be reduced and the lower bounds of model likelihood can be increased. Our calibration method is performed only once and the resulting models can be used repeatedly for sampling.
  arXiv  Detail & Related papers  (2023-02-21T14:14:40Z)
• Orthogonal SVD Covariance Conditioning and Latent Disentanglement [65.67315418971688]
  Inserting an SVD meta-layer into neural networks is prone to make the covariance ill-conditioned. We propose Nearest Orthogonal Gradient (NOG) and Optimal Learning Rate (OLR) Experiments on visual recognition demonstrate that our methods can simultaneously improve covariance conditioning and generalization.
  arXiv  Detail & Related papers  (2022-12-11T20:31:31Z)
• Optimization of Annealed Importance Sampling Hyperparameters [77.34726150561087]
  Annealed Importance Sampling (AIS) is a popular algorithm used to estimates the intractable marginal likelihood of deep generative models. We present a parameteric AIS process with flexible intermediary distributions and optimize the bridging distributions to use fewer number of steps for sampling. We assess the performance of our optimized AIS for marginal likelihood estimation of deep generative models and compare it to other estimators.
  arXiv  Detail & Related papers  (2022-09-27T07:58:25Z)
• Accelerating Diffusion Models via Early Stop of the Diffusion Process [114.48426684994179]
  Denoising Diffusion Probabilistic Models (DDPMs) have achieved impressive performance on various generation tasks. In practice DDPMs often need hundreds even thousands of denoising steps to obtain a high-quality sample. We propose a principled acceleration strategy, referred to as Early-Stopped DDPM (ES-DDPM), for DDPMs.
  arXiv  Detail & Related papers  (2022-05-25T06:40:09Z)
• Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models [39.11468968340014]
  Diffusion probabilistic models (DPMs) represent a class of powerful generative models. We propose Analytic-DPM, a training-free inference framework that estimates the analytic forms of the variance and KL divergence. We derive both lower and upper bounds of the optimal variance and clip the estimate for a better result.
  arXiv  Detail & Related papers  (2022-01-17T16:28:12Z)
• Learning to Efficiently Sample from Diffusion Probabilistic Models [49.58748345998702]
  Denoising Diffusion Probabilistic Models (DDPMs) can yield high-fidelity samples and competitive log-likelihoods across a range of domains. We introduce an exact dynamic programming algorithm that finds the optimal discrete time schedules for any pre-trained DDPM.
  arXiv  Detail & Related papers  (2021-06-07T17:15:07Z)
• Beyond the Mean-Field: Structured Deep Gaussian Processes Improve the Predictive Uncertainties [12.068153197381575]
  We propose a novel variational family that allows for retaining covariances between latent processes while achieving fast convergence. We provide an efficient implementation of our new approach and apply it to several benchmark datasets. It yields excellent results and strikes a better balance between accuracy and calibrated uncertainty estimates than its state-of-the-art alternatives.
  arXiv  Detail & Related papers  (2020-05-22T11:10:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.