Training Energy-Based Models with Diffusion Contrastive Divergences
- URL: http://arxiv.org/abs/2307.01668v1
- Date: Tue, 4 Jul 2023 12:00:06 GMT
- Title: Training Energy-Based Models with Diffusion Contrastive Divergences
- Abstract summary: Diffusion Contrastive Divergence (DCD) is a training objective for Energy-Based Models (EBMs)
We show that the proposed DCDs are both more computationally efficient than the CD and are not limited to a non-negligible gradient term.
In image generation experiments, the proposed DCD is capable of training an energy-based model for generating the Celab-A $32times 32$ dataset.
- Score: 45.01243513440672
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Energy-Based Models (EBMs) have been widely used for generative modeling.
Contrastive Divergence (CD), a prevailing training objective for EBMs, requires
sampling from the EBM with Markov Chain Monte Carlo methods (MCMCs), which
leads to an irreconcilable trade-off between the computational burden and the
validity of the CD. Running MCMCs till convergence is computationally
intensive. On the other hand, short-run MCMC brings in an extra non-negligible
parameter gradient term that is difficult to handle. In this paper, we provide
a general interpretation of CD, viewing it as a special instance of our
proposed Diffusion Contrastive Divergence (DCD) family. By replacing the
Langevin dynamic used in CD with other EBM-parameter-free diffusion processes,
we propose a more efficient divergence. We show that the proposed DCDs are both
more computationally efficient than the CD and are not limited to a
non-negligible gradient term. We conduct intensive experiments, including both
synthesis data modeling and high-dimensional image denoising and generation, to
show the advantages of the proposed DCDs. On the synthetic data learning and
image denoising experiments, our proposed DCD outperforms CD by a large margin.
In image generation experiments, the proposed DCD is capable of training an
energy-based model for generating the Celab-A $32\times 32$ dataset, which is
comparable to existing EBMs.
Related papers
- Shedding Light on Large Generative Networks: Estimating Epistemic Uncertainty in Diffusion Models [15.352556466952477]
Generative diffusion models are notable for their large parameter count (exceeding 100 million) and operation within high-dimensional image spaces.
We introduce an innovative framework, Diffusion Ensembles for Capturing Uncertainty (DECU), designed for estimating epistemic uncertainty for diffusion models.
arXiv Detail & Related papers (2024-06-05T14:03:21Z) - EM Distillation for One-step Diffusion Models [65.57766773137068]
We propose a maximum likelihood-based approach that distills a diffusion model to a one-step generator model with minimal loss of quality.
We develop a reparametrized sampling scheme and a noise cancellation technique that together stabilizes the distillation process.
arXiv Detail & Related papers (2024-05-27T05:55:22Z) - Distilling Diffusion Models into Conditional GANs [90.76040478677609]
We distill a complex multistep diffusion model into a single-step conditional GAN student model.
For efficient regression loss, we propose E-LatentLPIPS, a perceptual loss operating directly in diffusion model's latent space.
We demonstrate that our one-step generator outperforms cutting-edge one-step diffusion distillation models.
arXiv Detail & Related papers (2024-05-09T17:59:40Z) - Improving Adversarial Energy-Based Model via Diffusion Process [25.023967485839155]
Adversarial EBMs introduce a generator to form a minimax training game.
Inspired by diffusion-based models, we embedded EBMs into each denoising step to split a long-generated process into several smaller steps.
Our experiments show significant improvement in generation compared to existing adversarial EBMs.
arXiv Detail & Related papers (2024-03-04T01:33:53Z) - Generalized Contrastive Divergence: Joint Training of Energy-Based Model
and Diffusion Model through Inverse Reinforcement Learning [13.22531381403974]
Generalized Contrastive Divergence (GCD) is a novel objective function for training an energy-based model (EBM) and a sampler simultaneously.
We present preliminary yet promising results showing that joint training is beneficial for both EBM and a diffusion model.
arXiv Detail & Related papers (2023-12-06T10:10:21Z) - Learning Energy-Based Prior Model with Diffusion-Amortized MCMC [89.95629196907082]
Common practice of learning latent space EBMs with non-convergent short-run MCMC for prior and posterior sampling is hindering the model from further progress.
We introduce a simple but effective diffusion-based amortization method for long-run MCMC sampling and develop a novel learning algorithm for the latent space EBM based on it.
arXiv Detail & Related papers (2023-10-05T00:23:34Z) - Learning Energy-Based Models by Cooperative Diffusion Recovery Likelihood [64.95663299945171]
Training energy-based models (EBMs) on high-dimensional data can be both challenging and time-consuming.
There exists a noticeable gap in sample quality between EBMs and other generative frameworks like GANs and diffusion models.
We propose cooperative diffusion recovery likelihood (CDRL), an effective approach to tractably learn and sample from a series of EBMs.
arXiv Detail & Related papers (2023-09-10T22:05:24Z) - Pseudo-Spherical Contrastive Divergence [119.28384561517292]
We propose pseudo-spherical contrastive divergence (PS-CD) to generalize maximum learning likelihood of energy-based models.
PS-CD avoids the intractable partition function and provides a generalized family of learning objectives.
arXiv Detail & Related papers (2021-11-01T09:17:15Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.