Clarifying MCMC-based training of modern EBMs : Contrastive Divergence versus Maximum Likelihood

• URL: http://arxiv.org/abs/2202.12176v1
• Date: Thu, 24 Feb 2022 16:21:11 GMT
• Title: Clarifying MCMC-based training of modern EBMs : Contrastive Divergence versus Maximum Likelihood
• Abstract summary: The Energy-Based Model (EBM) framework is a very general approach to generative modeling that tries to learn and exploit probability distributions. We argue that the learning algorithm they use can in fact not be described as CD and reinterpret theirs methods in light of a new interpretation.
• Score: 2.7557703490816268
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The Energy-Based Model (EBM) framework is a very general approach to generative modeling that tries to learn and exploit probability distributions only defined though unnormalized scores. It has risen in popularity recently thanks to the impressive results obtained in image generation by parameterizing the distribution with Convolutional Neural Networks (CNN). However, the motivation and theoretical foundations behind modern EBMs are often absent from recent papers and this sometimes results in some confusion. In particular, the theoretical justifications behind the popular MCMC-based learning algorithm Contrastive Divergence (CD) are often glossed over and we find that this leads to theoretical errors in recent influential papers (Du & Mordatch, 2019; Du et al., 2020). After offering a first-principles introduction of MCMC-based training, we argue that the learning algorithm they use can in fact not be described as CD and reinterpret theirs methods in light of a new interpretation. Finally, we discuss the implications of our new interpretation and provide some illustrative experiments.

Related papers

• Information as Maximum-Caliber Deviation: A bridge between Integrated Information Theory and the Free Energy Principle [51.56484100374058]
  This work proposes that information can be defined as the deviation $$ of realized dynamics from a constrained maximum-caliber (MaxCal) path ensemble over a finite time horizon.<n>Under this definition, each of the cause/effect repertoires central to IIT 3.0 emerge directly from MaxCal variational principles.<n>Information $$ is shown to be equivalent to prediction error under accompanying predictive coding models.
  arXiv  Detail & Related papers  (2026-05-03T07:22:55Z)
• Learning Multi-Index Models with Hyper-Kernel Ridge Regression [9.500229343360491]
  We study the compositional structure of the learning task, namely the multi-index model (MIM)<n>Our main contribution is a sample complexity result demonstrating that HKRR can adaptively learn MIM, overcoming the curse of dimensionality.<n>We exploit the kernel nature of the estimator to develop ad hoc optimization approaches.
  arXiv  Detail & Related papers  (2025-10-02T20:03:33Z)
• Brewing Knowledge in Context: Distillation Perspectives on In-Context Learning [2.6129523822281415]
  In-context learning (ICL) allows large language models to solve novel tasks without weight updates.<n>Despite its empirical success, the mechanism behind ICL remains poorly understood.<n>We propose a new theoretical perspective that interprets ICL as an implicit form of knowledge distillation.
  arXiv  Detail & Related papers  (2025-06-13T07:17:41Z)
• Jarzynski Reweighting and Sampling Dynamics for Training Energy-Based Models: Theoretical Analysis of Different Transition Kernels [0.0]
  Energy-Based Models (EBMs) provide a flexible framework for generative modeling.<n>Traditional methods, such as contrastive divergence and score matching, introduce biases that can hinder accurate learning.<n>We present a theoretical analysis of Jarzynski reweighting, a technique from non-equilibrium statistical mechanics, and its implications for training EBMs.
  arXiv  Detail & Related papers  (2025-06-09T15:08:01Z)
• A PAC-Bayesian Perspective on the Interpolating Information Criterion [54.548058449535155]
  We show how a PAC-Bayes bound is obtained for a general class of models, characterizing factors which influence performance in the interpolating regime. We quantify how the test error for overparameterized models achieving effectively zero training error depends on the quality of the implicit regularization imposed by e.g. the combination of model, parameter-initialization scheme.
  arXiv  Detail & Related papers  (2023-11-13T01:48:08Z)
• Learning Energy-Based Prior Model with Diffusion-Amortized MCMC [89.95629196907082]
  Common practice of learning latent space EBMs with non-convergent short-run MCMC for prior and posterior sampling is hindering the model from further progress. We introduce a simple but effective diffusion-based amortization method for long-run MCMC sampling and develop a novel learning algorithm for the latent space EBM based on it.
  arXiv  Detail & Related papers  (2023-10-05T00:23:34Z)
• Sparse Linear Concept Discovery Models [11.138948381367133]
  Concept Bottleneck Models (CBMs) constitute a popular approach where hidden layers are tied to human understandable concepts. We propose a simple yet highly intuitive interpretable framework based on Contrastive Language Image models and a single sparse linear layer. We experimentally show, our framework not only outperforms recent CBM approaches accuracy-wise, but it also yields high per example concept sparsity.
  arXiv  Detail & Related papers  (2023-08-21T15:16:19Z)
• End-To-End Latent Variational Diffusion Models for Inverse Problems in High Energy Physics [61.44793171735013]
  We introduce a novel unified architecture, termed latent variation models, which combines the latent learning of cutting-edge generative art approaches with an end-to-end variational framework. Our unified approach achieves a distribution-free distance to the truth of over 20 times less than non-latent state-of-the-art baseline.
  arXiv  Detail & Related papers  (2023-05-17T17:43:10Z)
• A Mathematical Framework for Learning Probability Distributions [0.0]
  generative modeling and density estimation has become an immensely popular subject in recent years. This paper provides a mathematical framework such that all the well-known models can be derived based on simple principles. In particular, we prove that these models enjoy implicit regularization during training, so that the generalization error at early-stopping avoids the curse of dimensionality.
  arXiv  Detail & Related papers  (2022-12-22T04:41:45Z)
• Recent Advances in Large Margin Learning [63.982279380483526]
  This paper serves as a survey of recent advances in large margin training and its theoretical foundations, mostly for (nonlinear) deep neural networks (DNNs) We generalize the formulation of classification margins from classical research to latest DNNs, summarize theoretical connections between the margin, network generalization, and robustness, and introduce recent efforts in enlarging the margins for DNNs comprehensively.
  arXiv  Detail & Related papers  (2021-03-25T04:12:00Z)
• How to Train Your Energy-Based Models [19.65375049263317]
  Energy-Based Models (EBMs) specify probability density or mass functions up to an unknown normalizing constant. This tutorial is targeted at an audience with basic understanding of generative models who want to apply EBMs or start a research project in this direction.
  arXiv  Detail & Related papers  (2021-01-09T04:51:31Z)
• MixKD: Towards Efficient Distillation of Large-scale Language Models [129.73786264834894]
  We propose MixKD, a data-agnostic distillation framework, to endow the resulting model with stronger generalization ability. We prove from a theoretical perspective that under reasonable conditions MixKD gives rise to a smaller gap between the error and the empirical error. Experiments under a limited-data setting and ablation studies further demonstrate the advantages of the proposed approach.
  arXiv  Detail & Related papers  (2020-11-01T18:47:51Z)
• A Chain Graph Interpretation of Real-World Neural Networks [58.78692706974121]
  We propose an alternative interpretation that identifies NNs as chain graphs (CGs) and feed-forward as an approximate inference procedure. The CG interpretation specifies the nature of each NN component within the rich theoretical framework of probabilistic graphical models. We demonstrate with concrete examples that the CG interpretation can provide novel theoretical support and insights for various NN techniques.
  arXiv  Detail & Related papers  (2020-06-30T14:46:08Z)
• Demystifying Orthogonal Monte Carlo and Beyond [20.745014324028386]
  Orthogonal Monte Carlo (OMC) is a very effective sampling algorithm imposing structural geometric conditions (orthogonality) on samples for variance reduction. We shed new light on the theoretical principles behind OMC, applying theory of negatively dependent random variables to obtain several new concentration results. We propose a novel extensions of the method leveraging number theory techniques and particle algorithms, called Near-Orthogonal Monte Carlo (NOMC)
  arXiv  Detail & Related papers  (2020-05-27T18:44:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.