ED-VAE: Entropy Decomposition of ELBO in Variational Autoencoders

• URL: http://arxiv.org/abs/2407.06797v1
• Date: Tue, 9 Jul 2024 12:09:21 GMT
• Title: ED-VAE: Entropy Decomposition of ELBO in Variational Autoencoders
• Abstract summary: Variational Autoencoders (VAEs) are constrained by the limitations of the Evidence Lower Bound (ELBO) formulation. This work introduces the Entropy Decomposed Variational Autoencoder (ED-VAE), a novel re-formulation of the ELBO that explicitly includes entropy and cross-entropy components.
• Score: 0.850206009406913
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Traditional Variational Autoencoders (VAEs) are constrained by the limitations of the Evidence Lower Bound (ELBO) formulation, particularly when utilizing simplistic, non-analytic, or unknown prior distributions. These limitations inhibit the VAE's ability to generate high-quality samples and provide clear, interpretable latent representations. This work introduces the Entropy Decomposed Variational Autoencoder (ED-VAE), a novel re-formulation of the ELBO that explicitly includes entropy and cross-entropy components. This reformulation significantly enhances model flexibility, allowing for the integration of complex and non-standard priors. By providing more detailed control over the encoding and regularization of latent spaces, ED-VAE not only improves interpretability but also effectively captures the complex interactions between latent variables and observed data, thus leading to better generative performance.

Related papers

• eXact-Prior Variational Autoencoder (X-VAE): Learning Data-Adaptive Gaussian Mixture Priors for Latent Distributions [24.959038626924922]
  Variational Autoencoders (VAEs) commonly assume a standard isotropic Gaussian prior over the latent space.<n>We propose the eXact-Prior Variational Autoencoder (X-VAE), a framework that replaces the conventional standard normal prior with a Gaussian prior derived from the latent representations of a pretrained autoencoder (AE)<n> Experimental results demonstrate that X-VAE preserves reconstruction quality while producing latent representations that better align with the empirical data distribution.
  arXiv  Detail & Related papers  (2026-06-30T23:05:06Z)
• Entropic Auto-Encoding via Implicit Free-Energy Minimization [0.0]
  We introduce Entropic Autoencoders (EAEs), a framework in which reconstruction loss is the only explicit objective.<n>We demonstrate that EAEs mitigate posterior collapse by learning non-Gaussian, multimodal latent distributions.
  arXiv  Detail & Related papers  (2026-05-15T16:43:45Z)
• Instance-Adaptive Parametrization for Amortized Variational Inference [3.219880761967806]
  We propose the instance-adaptive variational autoencoder (IA-VAE)<n>IA-VAE is an amortized variational inference framework in which a hypernetwork generates input-dependent modulations of a shared encoder.<n>Experiments on synthetic data, where the true posterior is known, show that IA-VAE yields more accurate posterior approximations and reduces the amortization gap.
  arXiv  Detail & Related papers  (2026-04-08T08:11:38Z)
• Protect Before Generate: Error Correcting Codes within Discrete Deep Generative Models [3.053842954605396]
  We introduce a novel method that enhances variational inference in discrete latent variable models. We leverage Error Correcting Codes (ECCs) to introduce redundancy in the latent representations. This redundancy is then exploited by the variational posterior to yield more accurate estimates.
  arXiv  Detail & Related papers  (2024-10-10T11:59:58Z)
• A Non-negative VAE:the Generalized Gamma Belief Network [49.970917207211556]
  The gamma belief network (GBN) has demonstrated its potential for uncovering multi-layer interpretable latent representations in text data. We introduce the generalized gamma belief network (Generalized GBN) in this paper, which extends the original linear generative model to a more expressive non-linear generative model. We also propose an upward-downward Weibull inference network to approximate the posterior distribution of the latent variables.
  arXiv  Detail & Related papers  (2024-08-06T18:18:37Z)
• How to train your VAE [0.0]
  Variational Autoencoders (VAEs) have become a cornerstone in generative modeling and representation learning within machine learning. This paper explores interpreting the Kullback-Leibler (KL) Divergence, a critical component within the Evidence Lower Bound (ELBO) The proposed method redefines the ELBO with a mixture of Gaussians for the posterior probability, introduces a regularization term, and employs a PatchGAN discriminator to enhance texture realism.
  arXiv  Detail & Related papers  (2023-09-22T19:52:28Z)
• Curve Your Enthusiasm: Concurvity Regularization in Differentiable Generalized Additive Models [5.519653885553456]
  Generalized Additive Models (GAMs) have recently experienced a resurgence in popularity due to their interpretability. We show how concurvity can severly impair the interpretability of GAMs. We propose a remedy: a conceptually simple, yet effective regularizer which penalizes pairwise correlations of the non-linearly transformed feature variables.
  arXiv  Detail & Related papers  (2023-05-19T06:55:49Z)
• DIFFormer: Scalable (Graph) Transformers Induced by Energy Constrained Diffusion [66.21290235237808]
  We introduce an energy constrained diffusion model which encodes a batch of instances from a dataset into evolutionary states. We provide rigorous theory that implies closed-form optimal estimates for the pairwise diffusion strength among arbitrary instance pairs. Experiments highlight the wide applicability of our model as a general-purpose encoder backbone with superior performance in various tasks.
  arXiv  Detail & Related papers  (2023-01-23T15:18:54Z)
• Regularizing Variational Autoencoder with Diversity and Uncertainty Awareness [61.827054365139645]
  Variational Autoencoder (VAE) approximates the posterior of latent variables based on amortized variational inference. We propose an alternative model, DU-VAE, for learning a more Diverse and less Uncertain latent space.
  arXiv  Detail & Related papers  (2021-10-24T07:58:13Z)
• InteL-VAEs: Adding Inductive Biases to Variational Auto-Encoders via Intermediary Latents [60.785317191131284]
  We introduce a simple and effective method for learning VAEs with controllable biases by using an intermediary set of latent variables. In particular, it allows us to impose desired properties like sparsity or clustering on learned representations. We show that this, in turn, allows InteL-VAEs to learn both better generative models and representations.
  arXiv  Detail & Related papers  (2021-06-25T16:34:05Z)
• VAE Approximation Error: ELBO and Conditional Independence [78.72292013299868]
  This paper analyzes VAE approximation errors caused by the combination of the ELBO objective with the choice of the encoder probability family. We show that the ELBO subset can not be enlarged, and the respective error cannot be decreased, by only considering deeper encoder networks.
  arXiv  Detail & Related papers  (2021-02-18T12:54:42Z)
• Neural Decomposition: Functional ANOVA with Variational Autoencoders [9.51828574518325]
  Variational Autoencoders (VAEs) have become a popular approach for dimensionality reduction. Due to the black-box nature of VAEs, their utility for healthcare and genomics applications has been limited. We focus on characterising the sources of variation in Conditional VAEs.
  arXiv  Detail & Related papers  (2020-06-25T10:29:13Z)
• Stochastic Bottleneck: Rateless Auto-Encoder for Flexible Dimensionality Reduction [16.42183040840934]
  We propose a new concept of rateless auto-encoders (RL-AEs) that enable a flexible latent dimensionality. We experimentally demonstrate that the proposed RL-AEs can achieve variable dimensionality reduction while achieving low distortion.
  arXiv  Detail & Related papers  (2020-05-06T14:47:42Z)
• Improve Variational Autoencoder for Text Generationwith Discrete Latent Bottleneck [52.08901549360262]
  Variational autoencoders (VAEs) are essential tools in end-to-end representation learning. VAEs tend to ignore latent variables with a strong auto-regressive decoder. We propose a principled approach to enforce an implicit latent feature matching in a more compact latent space.
  arXiv  Detail & Related papers  (2020-04-22T14:41:37Z)
• Automatic Differentiation Variational Inference with Mixtures [4.995383193706478]
  We show how stratified sampling may be used to enable mixture distributions as the approximate posterior. We derive a new lower bound on the evidence analogous to the importance weighted autoencoder (IWAE)
  arXiv  Detail & Related papers  (2020-03-03T18:12:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.