Generative Models with ELBOs Converging to Entropy Sums

• URL: http://arxiv.org/abs/2501.09022v1
• Date: Wed, 25 Dec 2024 15:47:23 GMT
• Title: Generative Models with ELBOs Converging to Entropy Sums
• Authors: Jan Warnken, Dmytro Velychko, Simon Damm, Asja Fischer, Jörg Lücke,
• Abstract summary: Evidence lower bound (ELBO) is one of the most central objectives for unsupervised learning. We prove convergence to entropy sums for several generative models and model classes.
• Score: 12.962371546573229
• License:
• Abstract: The evidence lower bound (ELBO) is one of the most central objectives for probabilistic unsupervised learning. For the ELBOs of several generative models and model classes, we here prove convergence to entropy sums. As one result, we provide a list of generative models for which entropy convergence has been shown, so far, along with the corresponding expressions for entropy sums. Our considerations include very prominent generative models such as probabilistic PCA, sigmoid belief nets or Gaussian mixture models. However, we treat more models and entire model classes such as general mixtures of exponential family distributions. Our main contributions are the proofs for the individual models. For each given model we show that the conditions stated in Theorem 1 or Theorem 2 of [arXiv:2209.03077] are fulfilled such that by virtue of the theorems the given model's ELBO is equal to an entropy sum at all stationary points. The equality of the ELBO at stationary points applies under realistic conditions: for finite numbers of data points, for model/data mismatches, at any stationary point including saddle points etc, and it applies for any well behaved family of variational distributions.

Related papers

• von Mises Quasi-Processes for Bayesian Circular Regression [57.88921637944379]
  We explore a family of expressive and interpretable distributions over circle-valued random functions. The resulting probability model has connections with continuous spin models in statistical physics. For posterior inference, we introduce a new Stratonovich-like augmentation that lends itself to fast Markov Chain Monte Carlo sampling.
  arXiv  Detail & Related papers  (2024-06-19T01:57:21Z)
• Geometric Neural Diffusion Processes [55.891428654434634]
  We extend the framework of diffusion models to incorporate a series of geometric priors in infinite-dimension modelling. We show that with these conditions, the generative functional model admits the same symmetry.
  arXiv  Detail & Related papers  (2023-07-11T16:51:38Z)
• On the Convergence of the ELBO to Entropy Sums [3.345575993695074]
  We show that the variational lower bound is at all stationary points of learning equal to a sum of entropies. Proving equality of the ELBO to entropy sums at stationary points is the main contribution of this work.
  arXiv  Detail & Related papers  (2022-09-07T11:33:32Z)
• Sampling from Arbitrary Functions via PSD Models [55.41644538483948]
  We take a two-step approach by first modeling the probability distribution and then sampling from that model. We show that these models can approximate a large class of densities concisely using few evaluations, and present a simple algorithm to effectively sample from these models.
  arXiv  Detail & Related papers  (2021-10-20T12:25:22Z)
• PSD Representations for Effective Probability Models [117.35298398434628]
  We show that a recently proposed class of positive semi-definite (PSD) models for non-negative functions is particularly suited to this end. We characterize both approximation and generalization capabilities of PSD models, showing that they enjoy strong theoretical guarantees. Our results open the way to applications of PSD models to density estimation, decision theory and inference.
  arXiv  Detail & Related papers  (2021-06-30T15:13:39Z)
• On the Generative Utility of Cyclic Conditionals [103.1624347008042]
  We study whether and how can we model a joint distribution $p(x,z)$ using two conditional models $p(x|z)$ that form a cycle. We propose the CyGen framework for cyclic-conditional generative modeling, including methods to enforce compatibility and use the determined distribution to fit and generate data.
  arXiv  Detail & Related papers  (2021-06-30T10:23:45Z)
• Loss function based second-order Jensen inequality and its application to particle variational inference [112.58907653042317]
  Particle variational inference (PVI) uses an ensemble of models as an empirical approximation for the posterior distribution. PVI iteratively updates each model with a repulsion force to ensure the diversity of the optimized models. We derive a novel generalization error bound and show that it can be reduced by enhancing the diversity of models.
  arXiv  Detail & Related papers  (2021-06-09T12:13:51Z)
• Generalization and Memorization: The Bias Potential Model [9.975163460952045]
  generative models and density estimators behave quite differently from models for learning functions. For the bias potential model, we show that dimension-independent generalization accuracy is achievable if early stopping is adopted. In the long term, the model either memorizes the samples or diverges.
  arXiv  Detail & Related papers  (2020-11-29T04:04:54Z)
• Probability Link Models with Symmetric Information Divergence [1.5749416770494706]
  Two general classes of link models are proposed. The first model links two survival functions and is applicable to models such as the proportional odds and change point. The second model links two cumulative probability distribution functions.
  arXiv  Detail & Related papers  (2020-08-10T19:49:51Z)
• Eigenstate Entanglement Entropy in Random Quadratic Hamiltonians [0.0]
  In integrable models, the volume-law coefficient depends on the subsystem fraction. We show that the average entanglement entropy of eigenstates of the power-law random banded matrix model is close but not the same as the result for quadratic models.
  arXiv  Detail & Related papers  (2020-06-19T18:01:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.