Training Discrete Energy-Based Models with Energy Discrepancy

• URL: http://arxiv.org/abs/2307.07595v1
• Date: Fri, 14 Jul 2023 19:38:05 GMT
• Title: Training Discrete Energy-Based Models with Energy Discrepancy
• Authors: Tobias Schr\"oder, Zijing Ou, Yingzhen Li, Andrew B. Duncan
• Abstract summary: Training energy-based models (EBMs) on discrete spaces is challenging because sampling over such spaces can be difficult. We propose to train discrete EBMs with energy discrepancy (ED), a novel type of contrastive loss functional. We demonstrate their relative performance on lattice Ising models, binary synthetic data, and discrete image data sets.
• Score: 18.161764926125066
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Training energy-based models (EBMs) on discrete spaces is challenging because sampling over such spaces can be difficult. We propose to train discrete EBMs with energy discrepancy (ED), a novel type of contrastive loss functional which only requires the evaluation of the energy function at data points and their perturbed counter parts, thus not relying on sampling strategies like Markov chain Monte Carlo (MCMC). Energy discrepancy offers theoretical guarantees for a broad class of perturbation processes of which we investigate three types: perturbations based on Bernoulli noise, based on deterministic transforms, and based on neighbourhood structures. We demonstrate their relative performance on lattice Ising models, binary synthetic data, and discrete image data sets.

Related papers

• Neural Message Passing Induced by Energy-Constrained Diffusion [79.9193447649011]
  We propose an energy-constrained diffusion model as a principled interpretable framework for understanding the mechanism of MPNNs. We show that the new model can yield promising performance for cases where the data structures are observed (as a graph), partially observed or completely unobserved.
  arXiv  Detail & Related papers  (2024-09-13T17:54:41Z)
• Iterated Denoising Energy Matching for Sampling from Boltzmann Densities [109.23137009609519]
  Iterated Denoising Energy Matching (iDEM) iDEM alternates between (I) sampling regions of high model density from a diffusion-based sampler and (II) using these samples in our matching objective. We show that the proposed approach achieves state-of-the-art performance on all metrics and trains $2-5times$ faster.
  arXiv  Detail & Related papers  (2024-02-09T01:11:23Z)
• Energy-Based Models for Anomaly Detection: A Manifold Diffusion Recovery Approach [12.623417770432146]
  We present a new method of training energy-based models (EBMs) for anomaly detection that leverages low-dimensional structures within data. The proposed algorithm, Manifold Projection-Diffusion Recovery (MPDR), first perturbs a data point along a low-dimensional manifold that approximates the training dataset. Experimental results show that MPDR exhibits strong performance across various anomaly detection tasks involving diverse data types.
  arXiv  Detail & Related papers  (2023-10-28T11:18:39Z)
• Energy Discrepancies: A Score-Independent Loss for Energy-Based Models [20.250792836049882]
  We propose a novel loss function called Energy Discrepancy (ED) which does not rely on the computation of scores or expensive Markov chain Monte Carlo. We show that ED approaches the explicit score matching and negative log-likelihood loss under different limits, effectively interpolating between both.
  arXiv  Detail & Related papers  (2023-07-12T19:51:49Z)
• On Feature Diversity in Energy-based Models [98.78384185493624]
  An energy-based model (EBM) is typically formed of inner-model(s) that learn a combination of the different features to generate an energy mapping for each input configuration. We extend the probably approximately correct (PAC) theory of EBMs and analyze the effect of redundancy reduction on the performance of EBMs.
  arXiv  Detail & Related papers  (2023-06-02T12:30:42Z)
• Conjugate Energy-Based Models [7.8112101386166195]
  Conconjugate energy-based models (CEBMs) are a new class of energy-based models that define a joint density over data and latent variables. CEBMs have similar use cases as variational autoencoders, in the sense that they learn an unsupervised mapping from data to latent variables. Our experiments demonstrate that conjugate EBMs achieve competitive results in terms of image modelling, predictive power of latent space, and out-of-domain detection on a variety of datasets.
  arXiv  Detail & Related papers  (2021-06-25T17:51:41Z)
• Energy Aligning for Biased Models [39.00256193731365]
  Training on class-imbalanced data usually results in biased models that tend to predict samples into the majority classes. We propose a simple and effective method named Energy Aligning to eliminate the bias. Experimental results show that energy aligning can effectively alleviate class imbalance issue and outperform state-of-the-art methods on several benchmarks.
  arXiv  Detail & Related papers  (2021-06-07T05:12:26Z)
• Energy-Based Processes for Exchangeable Data [109.04978766553612]
  We introduce Energy-Based Processes (EBPs) to extend energy based models to exchangeable data. A key advantage of EBPs is the ability to express more flexible distributions over sets without restricting their cardinality. We develop an efficient training procedure for EBPs that demonstrates state-of-the-art performance on a variety of tasks.
  arXiv  Detail & Related papers  (2020-03-17T04:26:02Z)
• Training Deep Energy-Based Models with f-Divergence Minimization [113.97274898282343]
  Deep energy-based models (EBMs) are very flexible in distribution parametrization but computationally challenging. We propose a general variational framework termed f-EBM to train EBMs using any desired f-divergence. Experimental results demonstrate the superiority of f-EBM over contrastive divergence, as well as the benefits of training EBMs using f-divergences other than KL.
  arXiv  Detail & Related papers  (2020-03-06T23:11:13Z)
• Targeted free energy estimation via learned mappings [66.20146549150475]
  Free energy perturbation (FEP) was proposed by Zwanzig more than six decades ago as a method to estimate free energy differences. FEP suffers from a severe limitation: the requirement of sufficient overlap between distributions. One strategy to mitigate this problem, called Targeted Free Energy Perturbation, uses a high-dimensional mapping in configuration space to increase overlap.
  arXiv  Detail & Related papers  (2020-02-12T11:10:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.