Incorporating Inductive Biases to Energy-based Generative Models

• URL: http://arxiv.org/abs/2505.01111v1
• Date: Fri, 02 May 2025 08:46:03 GMT
• Title: Incorporating Inductive Biases to Energy-based Generative Models
• Authors: Yukun Li, Li-Ping Liu,
• Abstract summary: We introduce a novel hybrid approach that combines an EBM with an exponential family model to incorporate inductive bias into data modeling.<n> Specifically, we augment the energy term with a parameter-free statistic function to help the model capture key data statistics.<n>Our empirical study validates the hybrid model's ability to match statistics.
• Score: 5.140589325829964
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: With the advent of score-matching techniques for model training and Langevin dynamics for sample generation, energy-based models (EBMs) have gained renewed interest as generative models. Recent EBMs usually use neural networks to define their energy functions. In this work, we introduce a novel hybrid approach that combines an EBM with an exponential family model to incorporate inductive bias into data modeling. Specifically, we augment the energy term with a parameter-free statistic function to help the model capture key data statistics. Like an exponential family model, the hybrid model aims to align the distribution statistics with data statistics during model training, even when it only approximately maximizes the data likelihood. This property enables us to impose constraints on the hybrid model. Our empirical study validates the hybrid model's ability to match statistics. Furthermore, experimental results show that data fitting and generation improve when suitable informative statistics are incorporated into the hybrid model.

Related papers

• Guiding Time-Varying Generative Models with Natural Gradients on Exponential Family Manifold [5.000311680307273]
  We show that the evolution of time-varying generative models can be projected onto an exponential family manifold.<n>We then train the generative model by moving its projection on the manifold according to the natural gradient descent scheme.<n>We propose particle versions of the algorithm, which feature closed-form update rules for any parametric model within the exponential family.
  arXiv  Detail & Related papers  (2025-02-11T15:39:47Z)
• Energy-Based Diffusion Language Models for Text Generation [126.23425882687195]
  Energy-based Diffusion Language Model (EDLM) is an energy-based model operating at the full sequence level for each diffusion step.<n>Our framework offers a 1.3$times$ sampling speedup over existing diffusion models.
  arXiv  Detail & Related papers  (2024-10-28T17:25:56Z)
• Maximum Entropy Inverse Reinforcement Learning of Diffusion Models with Energy-Based Models [12.327318533784961]
  We present a maximum reinforcement learning (IRL) approach for improving the sample quality of diffusion generative models. We train (or fine-tune) a diffusion model using the log density estimated from training data. Our empirical studies show that diffusion models fine-tuned using DxMI can generate high-quality samples in as few as 4 and 10 steps.
  arXiv  Detail & Related papers  (2024-06-30T08:52:17Z)
• Towards Theoretical Understandings of Self-Consuming Generative Models [56.84592466204185]
  This paper tackles the emerging challenge of training generative models within a self-consuming loop. We construct a theoretical framework to rigorously evaluate how this training procedure impacts the data distributions learned by future models. We present results for kernel density estimation, delivering nuanced insights such as the impact of mixed data training on error propagation.
  arXiv  Detail & Related papers  (2024-02-19T02:08:09Z)
• On the Stability of Iterative Retraining of Generative Models on their own Data [56.153542044045224]
  We study the impact of training generative models on mixed datasets. We first prove the stability of iterative training under the condition that the initial generative models approximate the data distribution well enough. We empirically validate our theory on both synthetic and natural images by iteratively training normalizing flows and state-of-the-art diffusion models.
  arXiv  Detail & Related papers  (2023-09-30T16:41:04Z)
• Your Autoregressive Generative Model Can be Better If You Treat It as an Energy-Based One [83.5162421521224]
  We propose a unique method termed E-ARM for training autoregressive generative models. E-ARM takes advantage of a well-designed energy-based learning objective. We show that E-ARM can be trained efficiently and is capable of alleviating the exposure bias problem.
  arXiv  Detail & Related papers  (2022-06-26T10:58:41Z)
• Hybrid Feature- and Similarity-Based Models for Prediction and Interpretation using Large-Scale Observational Data [0.0]
  We propose a hybrid feature- and similarity-based model for supervised learning. The proposed hybrid model is fit by convex optimization with a sparsity-inducing penalty on the kernel portion. We compared our models to solely feature- and similarity-based approaches using synthetic data and using EHR data to predict risk of loneliness or social isolation.
  arXiv  Detail & Related papers  (2022-04-12T20:37:03Z)
• Mixed Effects Neural ODE: A Variational Approximation for Analyzing the Dynamics of Panel Data [50.23363975709122]
  We propose a probabilistic model called ME-NODE to incorporate (fixed + random) mixed effects for analyzing panel data. We show that our model can be derived using smooth approximations of SDEs provided by the Wong-Zakai theorem. We then derive Evidence Based Lower Bounds for ME-NODE, and develop (efficient) training algorithms.
  arXiv  Detail & Related papers  (2022-02-18T22:41:51Z)
• Controllable and Compositional Generation with Latent-Space Energy-Based Models [60.87740144816278]
  Controllable generation is one of the key requirements for successful adoption of deep generative models in real-world applications. In this work, we use energy-based models (EBMs) to handle compositional generation over a set of attributes. By composing energy functions with logical operators, this work is the first to achieve such compositionality in generating photo-realistic images of resolution 1024x1024.
  arXiv  Detail & Related papers  (2021-10-21T03:31:45Z)
• On Energy-Based Models with Overparametrized Shallow Neural Networks [44.74000986284978]
  Energy-based models (EBMs) are a powerful framework for generative modeling. In this work we focus on shallow neural networks. We show that models trained in the so-called "active" regime provide a statistical advantage over their associated "lazy" or kernel regime.
  arXiv  Detail & Related papers  (2021-04-15T15:34:58Z)
• Developing a Hybrid Data-Driven, Mechanistic Virtual Flow Meter -- a Case Study [0.0]
  This research investigates a hybrid modeling approach, utilizing techniques from both the aforementioned areas of expertise, to model a well production choke. The choke is represented with a simplified set of first-principle equations and a neural network to estimate the valve flow coefficient.
  arXiv  Detail & Related papers  (2020-02-07T12:35:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.