Preventing Posterior Collapse Induced by Oversmoothing in Gaussian VAE

• URL: http://arxiv.org/abs/2102.08663v1
• Date: Wed, 17 Feb 2021 10:00:49 GMT
• Title: Preventing Posterior Collapse Induced by Oversmoothing in Gaussian VAE
• Authors: Yuhta Takida, Wei-Hsiang Liao, Toshimitsu Uesaka, Shusuke Takahashi and Yuki Mitsufuji
• Abstract summary: We propose AR-ELBO, which controls the smoothness of the model by adapting the variance parameter. In addition, we extend VAE with alternative parameterizations on the variance parameter to deal with non-uniform or conditional data variance. The proposed VAE extensions trained with AR-ELBO show improved Fr'echet inception distance (FID) on images generated from the MNIST and CelebA datasets.
• Score: 7.845959449872641
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Variational autoencoders (VAEs) often suffer from posterior collapse, which is a phenomenon in which the learned latent space becomes uninformative. This is often related to a hyperparameter resembling the data variance. It can be shown that an inappropriate choice of this parameter causes oversmoothness and leads to posterior collapse in the linearly approximated case and can be empirically verified for the general cases. Therefore, we propose AR-ELBO (Adaptively Regularized Evidence Lower BOund), which controls the smoothness of the model by adapting this variance parameter. In addition, we extend VAE with alternative parameterizations on the variance parameter to deal with non-uniform or conditional data variance. The proposed VAE extensions trained with AR-ELBO show improved Fr\'echet inception distance (FID) on images generated from the MNIST and CelebA datasets.

Related papers

• Beyond Diagonal Covariance: Flexible Posterior VAEs via Free-Form Injective Flows [7.655073661577189]
  Variational Autoencoders (VAEs) are powerful generative models widely used for learning interpretable latent spaces, quantifying uncertainty, and compressing data for downstream generative tasks.<n>We show that a regularized variant of the recently introduced Free-form Injective Flow (FIF) can be interpreted as a VAE featuring a highly flexible, implicitly defined posterior.
  arXiv  Detail & Related papers  (2025-06-02T10:36:27Z)
• Improved Anomaly Detection through Conditional Latent Space VAE Ensembles [49.1574468325115]
  Conditional Latent space Variational Autoencoder (CL-VAE) improved pre-processing for anomaly detection on data with known inlier classes and unknown outlier classes. Model shows increased accuracy in anomaly detection, achieving an AUC of 97.4% on the MNIST dataset. In addition, the CL-VAE shows increased benefits from ensembling, a more interpretable latent space, and an increased ability to learn patterns in complex data with limited model sizes.
  arXiv  Detail & Related papers  (2024-10-16T07:48:53Z)
• Matching aggregate posteriors in the variational autoencoder [0.5759862457142761]
  The variational autoencoder (VAE) is a well-studied, deep, latent-variable model (DLVM) This paper addresses shortcomings in VAEs by reformulating the objective function associated with VAEs in order to match the aggregate/marginal posterior distribution to the prior. The proposed method is named the emphaggregate variational autoencoder (AVAE) and is built on the theoretical framework of the VAE.
  arXiv  Detail & Related papers  (2023-11-13T19:22:37Z)
• Winning Prize Comes from Losing Tickets: Improve Invariant Learning by Exploring Variant Parameters for Out-of-Distribution Generalization [76.27711056914168]
  Out-of-Distribution (OOD) Generalization aims to learn robust models that generalize well to various environments without fitting to distribution-specific features. Recent studies based on Lottery Ticket Hypothesis (LTH) address this problem by minimizing the learning target to find some of the parameters that are critical to the task. We propose Exploring Variant parameters for Invariant Learning (EVIL) which also leverages the distribution knowledge to find the parameters that are sensitive to distribution shift.
  arXiv  Detail & Related papers  (2023-10-25T06:10:57Z)
• DAVA: Disentangling Adversarial Variational Autoencoder [12.513372993000914]
  We introduce DAVA, a novel training procedure for variational auto-encoders. We demonstrate the ability of PIPE to positively predict the performance of downstream models in abstract reasoning.
  arXiv  Detail & Related papers  (2023-03-02T16:08:23Z)
• Posterior Collapse and Latent Variable Non-identifiability [54.842098835445]
  We propose a class of latent-identifiable variational autoencoders, deep generative models which enforce identifiability without sacrificing flexibility. Across synthetic and real datasets, latent-identifiable variational autoencoders outperform existing methods in mitigating posterior collapse and providing meaningful representations of the data.
  arXiv  Detail & Related papers  (2023-01-02T06:16:56Z)
• DPVIm: Differentially Private Variational Inference Improved [13.761202518891329]
  Differentially private (DP) release of multidimensional statistics typically considers an aggregate sensitivity. Different dimensions of that vector might have widely different magnitudes and therefore DP perturbation disproportionately affects the signal across dimensions. We observe this problem in the gradient release of the DP-SGD algorithm when using it for variational inference (VI)
  arXiv  Detail & Related papers  (2022-10-28T07:41:32Z)
• Equivariance Discovery by Learned Parameter-Sharing [153.41877129746223]
  We study how to discover interpretable equivariances from data. Specifically, we formulate this discovery process as an optimization problem over a model's parameter-sharing schemes. Also, we theoretically analyze the method for Gaussian data and provide a bound on the mean squared gap between the studied discovery scheme and the oracle scheme.
  arXiv  Detail & Related papers  (2022-04-07T17:59:19Z)
• Learning Invariances in Neural Networks [51.20867785006147]
  We show how to parameterize a distribution over augmentations and optimize the training loss simultaneously with respect to the network parameters and augmentation parameters. We can recover the correct set and extent of invariances on image classification, regression, segmentation, and molecular property prediction from a large space of augmentations.
  arXiv  Detail & Related papers  (2020-10-22T17:18:48Z)
• Addressing Variance Shrinkage in Variational Autoencoders using Quantile Regression [0.0]
  Probable Variational AutoEncoder (VAE) has become a popular model for anomaly detection in applications such as lesion detection in medical images. We describe an alternative approach that avoids the well-known problem of shrinkage or underestimation of variance. Using estimated quantiles to compute mean and variance under the Gaussian assumption, we compute reconstruction probability as a principled approach to outlier or anomaly detection.
  arXiv  Detail & Related papers  (2020-10-18T17:37:39Z)
• Preventing Posterior Collapse with Levenshtein Variational Autoencoder [61.30283661804425]
  We propose to replace the evidence lower bound (ELBO) with a new objective which is simple to optimize and prevents posterior collapse. We show that Levenstein VAE produces more informative latent representations than alternative approaches to preventing posterior collapse.
  arXiv  Detail & Related papers  (2020-04-30T13:27:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.