How to Train Your Energy-Based Models

• URL: http://arxiv.org/abs/2101.03288v2
• Date: Wed, 17 Feb 2021 19:20:09 GMT
• Title: How to Train Your Energy-Based Models
• Authors: Yang Song and Diederik P. Kingma
• Abstract summary: Energy-Based Models (EBMs) specify probability density or mass functions up to an unknown normalizing constant. This tutorial is targeted at an audience with basic understanding of generative models who want to apply EBMs or start a research project in this direction.
• Score: 19.65375049263317
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Energy-Based Models (EBMs), also known as non-normalized probabilistic models, specify probability density or mass functions up to an unknown normalizing constant. Unlike most other probabilistic models, EBMs do not place a restriction on the tractability of the normalizing constant, thus are more flexible to parameterize and can model a more expressive family of probability distributions. However, the unknown normalizing constant of EBMs makes training particularly difficult. Our goal is to provide a friendly introduction to modern approaches for EBM training. We start by explaining maximum likelihood training with Markov chain Monte Carlo (MCMC), and proceed to elaborate on MCMC-free approaches, including Score Matching (SM) and Noise Constrastive Estimation (NCE). We highlight theoretical connections among these three approaches, and end with a brief survey on alternative training methods, which are still under active research. Our tutorial is targeted at an audience with basic understanding of generative models who want to apply EBMs or start a research project in this direction.

Related papers

• Hitchhiker's guide on Energy-Based Models: a comprehensive review on the relation with other generative models, sampling and statistical physics [0.0]
  Energy-Based Models (EBMs) have emerged as a powerful framework in the realm of generative modeling. This review aims to provide physicists with a comprehensive understanding of EBMs, delineating their connection to other generative models.
  arXiv  Detail & Related papers  (2024-06-19T16:08:00Z)
• Learning Energy-based Model via Dual-MCMC Teaching [5.31573596283377]
  Learning the energy-based model (EBM) can be achieved using the maximum likelihood estimation (MLE) This paper studies the fundamental learning problem of the energy-based model (EBM)
  arXiv  Detail & Related papers  (2023-12-05T03:39:54Z)
• Learning Energy-Based Prior Model with Diffusion-Amortized MCMC [89.95629196907082]
  Common practice of learning latent space EBMs with non-convergent short-run MCMC for prior and posterior sampling is hindering the model from further progress. We introduce a simple but effective diffusion-based amortization method for long-run MCMC sampling and develop a novel learning algorithm for the latent space EBM based on it.
  arXiv  Detail & Related papers  (2023-10-05T00:23:34Z)
• Learning Energy-Based Models by Cooperative Diffusion Recovery Likelihood [64.95663299945171]
  Training energy-based models (EBMs) on high-dimensional data can be both challenging and time-consuming. There exists a noticeable gap in sample quality between EBMs and other generative frameworks like GANs and diffusion models. We propose cooperative diffusion recovery likelihood (CDRL), an effective approach to tractably learn and sample from a series of EBMs.
  arXiv  Detail & Related papers  (2023-09-10T22:05:24Z)
• Guiding Energy-based Models via Contrastive Latent Variables [81.68492940158436]
  An energy-based model (EBM) is a popular generative framework that offers both explicit density and architectural flexibility. There often exists a large gap between EBMs and other generative frameworks like GANs in terms of generation quality. We propose a novel and effective framework for improving EBMs via contrastive representation learning.
  arXiv  Detail & Related papers  (2023-03-06T10:50:25Z)
• Mitigating Out-of-Distribution Data Density Overestimation in Energy-Based Models [54.06799491319278]
  Deep energy-based models (EBMs) are receiving increasing attention due to their ability to learn complex distributions. To train deep EBMs, the maximum likelihood estimation (MLE) with short-run Langevin Monte Carlo (LMC) is often used. We investigate why the MLE with short-run LMC can converge to EBMs with wrong density estimates.
  arXiv  Detail & Related papers  (2022-05-30T02:49:17Z)
• Learning Energy-Based Model with Variational Auto-Encoder as Amortized Sampler [35.80109055748496]
  Training energy-based models (EBMs) by maximum likelihood requires Markov chain Monte Carlo sampling. We learn a variational auto-encoder (VAE) to initialize the finite-step MCMC, such as Langevin dynamics that is derived from the energy function. With these amortized MCMC samples, the EBM can be trained by maximum likelihood, which follows an "analysis by synthesis" scheme. We call this joint training algorithm the variational MCMC teaching, in which the VAE chases the EBM toward data distribution.
  arXiv  Detail & Related papers  (2020-12-29T20:46:40Z)
• No MCMC for me: Amortized sampling for fast and stable training of energy-based models [62.1234885852552]
  Energy-Based Models (EBMs) present a flexible and appealing way to represent uncertainty. We present a simple method for training EBMs at scale using an entropy-regularized generator to amortize the MCMC sampling. Next, we apply our estimator to the recently proposed Joint Energy Model (JEM), where we match the original performance with faster and stable training.
  arXiv  Detail & Related papers  (2020-10-08T19:17:20Z)
• MCMC Should Mix: Learning Energy-Based Model with Neural Transport Latent Space MCMC [110.02001052791353]
  Learning energy-based model (EBM) requires MCMC sampling of the learned model as an inner loop of the learning algorithm. We show that the model has a particularly simple form in the space of the latent variables of the backbone model.
  arXiv  Detail & Related papers  (2020-06-12T01:25:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.