Learning Proposals for Practical Energy-Based Regression

• URL: http://arxiv.org/abs/2110.11948v2
• Date: Tue, 7 Nov 2023 11:23:19 GMT
• Title: Learning Proposals for Practical Energy-Based Regression
• Authors: Fredrik K. Gustafsson, Martin Danelljan, Thomas B. Sch\"on
• Abstract summary: Energy-based models (EBMs) have experienced a resurgence within machine learning in recent years. We introduce a conceptually simple method to automatically learn an effective proposal distribution, which is parameterized by a separate network head. At test-time, we can then employ importance sampling with the trained proposal to efficiently evaluate the learned EBM and produce stand-alone predictions.
• Score: 46.05502630457458
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Energy-based models (EBMs) have experienced a resurgence within machine learning in recent years, including as a promising alternative for probabilistic regression. However, energy-based regression requires a proposal distribution to be manually designed for training, and an initial estimate has to be provided at test-time. We address both of these issues by introducing a conceptually simple method to automatically learn an effective proposal distribution, which is parameterized by a separate network head. To this end, we derive a surprising result, leading to a unified training objective that jointly minimizes the KL divergence from the proposal to the EBM, and the negative log-likelihood of the EBM. At test-time, we can then employ importance sampling with the trained proposal to efficiently evaluate the learned EBM and produce stand-alone predictions. Furthermore, we utilize our derived training objective to learn mixture density networks (MDNs) with a jointly trained energy-based teacher, consistently outperforming conventional MDN training on four real-world regression tasks within computer vision. Code is available at https://github.com/fregu856/ebms_proposals.

Related papers

• LSEBMCL: A Latent Space Energy-Based Model for Continual Learning [20.356926275395004]
  The study demonstrates the efficacy of EBM in NLP tasks, achieving state-of-the-art results in all experiments. The proposed solution LSEBMCL (Latent Space Energy-Based Model for Continual Learning) in this work is to use energy-based models (EBMs) to prevent catastrophic forgetting.
  arXiv  Detail & Related papers  (2025-01-09T15:47:30Z)
• Learning Energy-Based Models by Cooperative Diffusion Recovery Likelihood [64.95663299945171]
  Training energy-based models (EBMs) on high-dimensional data can be both challenging and time-consuming. There exists a noticeable gap in sample quality between EBMs and other generative frameworks like GANs and diffusion models. We propose cooperative diffusion recovery likelihood (CDRL), an effective approach to tractably learn and sample from a series of EBMs.
  arXiv  Detail & Related papers  (2023-09-10T22:05:24Z)
• Non-Generative Energy Based Models [3.1447898427012473]
  Energy-based models (EBM) have become increasingly popular within computer vision. We propose a non-generative training approach, Non-Generative EBM (NG-EBM) We show that our NG-EBM training strategy retains many of the benefits of EBM in calibration, out-of-distribution detection, and adversarial resistance.
  arXiv  Detail & Related papers  (2023-04-03T18:47:37Z)
• CoopInit: Initializing Generative Adversarial Networks via Cooperative Learning [50.90384817689249]
  CoopInit is a cooperative learning-based strategy that can quickly learn a good starting point for GANs. We demonstrate the effectiveness of the proposed approach on image generation and one-sided unpaired image-to-image translation tasks.
  arXiv  Detail & Related papers  (2023-03-21T07:49:32Z)
• Energy-Efficient and Federated Meta-Learning via Projected Stochastic Gradient Ascent [79.58680275615752]
  We propose an energy-efficient federated meta-learning framework. We assume each task is owned by a separate agent, so a limited number of tasks is used to train a meta-model.
  arXiv  Detail & Related papers  (2021-05-31T08:15:44Z)
• Federated Learning for Short-term Residential Energy Demand Forecasting [4.769747792846004]
  Energy demand forecasting is an essential task performed within the energy industry to help balance supply with demand and maintain a stable load on the electricity grid. As supply transitions towards less reliable renewable energy generation, smart meters will prove a vital component to aid these forecasting tasks. However, smart meter take-up is low among privacy-conscious consumers that fear intrusion upon their fine-grained consumption data.
  arXiv  Detail & Related papers  (2021-05-27T17:33:09Z)
• No MCMC for me: Amortized sampling for fast and stable training of energy-based models [62.1234885852552]
  Energy-Based Models (EBMs) present a flexible and appealing way to represent uncertainty. We present a simple method for training EBMs at scale using an entropy-regularized generator to amortize the MCMC sampling. Next, we apply our estimator to the recently proposed Joint Energy Model (JEM), where we match the original performance with faster and stable training.
  arXiv  Detail & Related papers  (2020-10-08T19:17:20Z)
• Adaptive Serverless Learning [114.36410688552579]
  We propose a novel adaptive decentralized training approach, which can compute the learning rate from data dynamically. Our theoretical results reveal that the proposed algorithm can achieve linear speedup with respect to the number of workers. To reduce the communication-efficient overhead, we further propose a communication-efficient adaptive decentralized training approach.
  arXiv  Detail & Related papers  (2020-08-24T13:23:02Z)
• Energy-Based Imitation Learning [29.55675131809474]
  We tackle a common scenario in imitation learning (IL) where agents try to recover the optimal policy from expert demonstrations. Inspired by recent progress in energy-based model (EBM), in this paper we propose a simplified IL framework named Energy-Based Imitation Learning (EBIL) EBIL combines the idea of both EBM and occupancy measure matching, and via theoretic analysis we reveal that EBIL and Max-Entropy IRL (MaxEnt IRL) approaches are two sides of the same coin.
  arXiv  Detail & Related papers  (2020-04-20T15:49:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.