Generalized Contrastive Divergence: Joint Training of Energy-Based Model and Diffusion Model through Inverse Reinforcement Learning

• URL: http://arxiv.org/abs/2312.03397v1
• Date: Wed, 6 Dec 2023 10:10:21 GMT
• Title: Generalized Contrastive Divergence: Joint Training of Energy-Based Model and Diffusion Model through Inverse Reinforcement Learning
• Authors: Sangwoong Yoon, Dohyun Kwon, Himchan Hwang, Yung-Kyun Noh, Frank C. Park
• Abstract summary: Generalized Contrastive Divergence (GCD) is a novel objective function for training an energy-based model (EBM) and a sampler simultaneously. We present preliminary yet promising results showing that joint training is beneficial for both EBM and a diffusion model.
• Score: 13.22531381403974
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: We present Generalized Contrastive Divergence (GCD), a novel objective function for training an energy-based model (EBM) and a sampler simultaneously. GCD generalizes Contrastive Divergence (Hinton, 2002), a celebrated algorithm for training EBM, by replacing Markov Chain Monte Carlo (MCMC) distribution with a trainable sampler, such as a diffusion model. In GCD, the joint training of EBM and a diffusion model is formulated as a minimax problem, which reaches an equilibrium when both models converge to the data distribution. The minimax learning with GCD bears interesting equivalence to inverse reinforcement learning, where the energy corresponds to a negative reward, the diffusion model is a policy, and the real data is expert demonstrations. We present preliminary yet promising results showing that joint training is beneficial for both EBM and a diffusion model. GCD enables EBM training without MCMC while improving the sample quality of a diffusion model.

Related papers

• Bellman Diffusion: Generative Modeling as Learning a Linear Operator in the Distribution Space [72.52365911990935]
  We introduce Bellman Diffusion, a novel DGM framework that maintains linearity in MDPs through gradient and scalar field modeling. Our results show that Bellman Diffusion achieves accurate field estimations and is a capable image generator, converging 1.5x faster than the traditional histogram-based baseline in distributional RL tasks.
  arXiv  Detail & Related papers  (2024-10-02T17:53:23Z)
• Maximum Entropy Inverse Reinforcement Learning of Diffusion Models with Energy-Based Models [12.327318533784961]
  We present a maximum reinforcement learning (IRL) approach for improving the sample quality of diffusion generative models. We train (or fine-tune) a diffusion model using the log density estimated from training data. Our empirical studies show that diffusion models fine-tuned using DxMI can generate high-quality samples in as few as 4 and 10 steps.
  arXiv  Detail & Related papers  (2024-06-30T08:52:17Z)
• Diffusion Model Conditioning on Gaussian Mixture Model and Negative Gaussian Mixture Gradient [1.9298401192674903]
  Diffusion models (DMs) are a type of generative model that has a huge impact on image synthesis and beyond. We propose a conditioning mechanism utilizing Gaussian mixture models (GMMs) as feature conditioning to guide the denoising process. We show that conditional latent distribution based on features and classes is significantly different, so that conditional latent distribution on features produces fewer defect generations than conditioning on classes.
  arXiv  Detail & Related papers  (2024-01-20T16:01:18Z)
• Learning Energy-Based Prior Model with Diffusion-Amortized MCMC [89.95629196907082]
  Common practice of learning latent space EBMs with non-convergent short-run MCMC for prior and posterior sampling is hindering the model from further progress. We introduce a simple but effective diffusion-based amortization method for long-run MCMC sampling and develop a novel learning algorithm for the latent space EBM based on it.
  arXiv  Detail & Related papers  (2023-10-05T00:23:34Z)
• Learning Energy-Based Models by Cooperative Diffusion Recovery Likelihood [64.95663299945171]
  Training energy-based models (EBMs) on high-dimensional data can be both challenging and time-consuming. There exists a noticeable gap in sample quality between EBMs and other generative frameworks like GANs and diffusion models. We propose cooperative diffusion recovery likelihood (CDRL), an effective approach to tractably learn and sample from a series of EBMs.
  arXiv  Detail & Related papers  (2023-09-10T22:05:24Z)
• Diff-Instruct: A Universal Approach for Transferring Knowledge From Pre-trained Diffusion Models [77.83923746319498]
  We propose a framework called Diff-Instruct to instruct the training of arbitrary generative models. We show that Diff-Instruct results in state-of-the-art single-step diffusion-based models. Experiments on refining GAN models show that the Diff-Instruct can consistently improve the pre-trained generators of GAN models.
  arXiv  Detail & Related papers  (2023-05-29T04:22:57Z)
• Persistently Trained, Diffusion-assisted Energy-based Models [18.135784288023928]
  We introduce diffusion data and learn a joint EBM, called diffusion assisted-EBMs, through persistent training. We show that persistently trained EBMs can simultaneously achieve long-run stability, post-training image generation, and superior out-of-distribution detection.
  arXiv  Detail & Related papers  (2023-04-21T02:29:18Z)
• Particle Dynamics for Learning EBMs [83.59335980576637]
  Energy-based modeling is a promising approach to unsupervised learning, which yields many downstream applications from a single model. The main difficulty in learning energy-based models with the "contrastive approaches" is the generation of samples from the current energy function at each iteration. This paper proposes an alternative approach to getting these samples and avoiding crude MCMC sampling from the current model.
  arXiv  Detail & Related papers  (2021-11-26T23:41:07Z)
• Learning Energy-Based Model with Variational Auto-Encoder as Amortized Sampler [35.80109055748496]
  Training energy-based models (EBMs) by maximum likelihood requires Markov chain Monte Carlo sampling. We learn a variational auto-encoder (VAE) to initialize the finite-step MCMC, such as Langevin dynamics that is derived from the energy function. With these amortized MCMC samples, the EBM can be trained by maximum likelihood, which follows an "analysis by synthesis" scheme. We call this joint training algorithm the variational MCMC teaching, in which the VAE chases the EBM toward data distribution.
  arXiv  Detail & Related papers  (2020-12-29T20:46:40Z)
• Training Deep Energy-Based Models with f-Divergence Minimization [113.97274898282343]
  Deep energy-based models (EBMs) are very flexible in distribution parametrization but computationally challenging. We propose a general variational framework termed f-EBM to train EBMs using any desired f-divergence. Experimental results demonstrate the superiority of f-EBM over contrastive divergence, as well as the benefits of training EBMs using f-divergences other than KL.
  arXiv  Detail & Related papers  (2020-03-06T23:11:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.