Simple and Effective VAE Training with Calibrated Decoders

• URL: http://arxiv.org/abs/2006.13202v3
• Date: Mon, 12 Jul 2021 04:06:41 GMT
• Title: Simple and Effective VAE Training with Calibrated Decoders
• Authors: Oleh Rybkin, Kostas Daniilidis, Sergey Levine
• Abstract summary: Variational autoencoders (VAEs) provide an effective and simple method for modeling complex distributions. We study the impact of calibrated decoders, which learn the uncertainty of the decoding distribution. We propose a simple but novel modification to the commonly used Gaussian decoder, which computes the prediction variance analytically.
• Score: 123.08908889310258
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Variational autoencoders (VAEs) provide an effective and simple method for modeling complex distributions. However, training VAEs often requires considerable hyperparameter tuning to determine the optimal amount of information retained by the latent variable. We study the impact of calibrated decoders, which learn the uncertainty of the decoding distribution and can determine this amount of information automatically, on the VAE performance. While many methods for learning calibrated decoders have been proposed, many of the recent papers that employ VAEs rely on heuristic hyperparameters and ad-hoc modifications instead. We perform the first comprehensive comparative analysis of calibrated decoder and provide recommendations for simple and effective VAE training. Our analysis covers a range of image and video datasets and several single-image and sequential VAE models. We further propose a simple but novel modification to the commonly used Gaussian decoder, which computes the prediction variance analytically. We observe empirically that using heuristic modifications is not necessary with our method. Project website is at https://orybkin.github.io/sigma-vae/

Related papers

• Symmetric Equilibrium Learning of VAEs [56.56929742714685]
  We view variational autoencoders (VAEs) as decoder-encoder pairs, which map distributions in the data space to distributions in the latent space and vice versa. We propose a Nash equilibrium learning approach, which is symmetric with respect to the encoder and decoder and allows learning VAEs in situations where both the data and the latent distributions are accessible only by sampling.
  arXiv  Detail & Related papers  (2023-07-19T10:27:34Z)
• Fully Bayesian Autoencoders with Latent Sparse Gaussian Processes [23.682509357305406]
  Autoencoders and their variants are among the most widely used models in representation learning and generative modeling. We propose a novel Sparse Gaussian Process Bayesian Autoencoder model in which we impose fully sparse Gaussian Process priors on the latent space of a Bayesian Autoencoder.
  arXiv  Detail & Related papers  (2023-02-09T09:57:51Z)
• String-based Molecule Generation via Multi-decoder VAE [56.465033997245776]
  We investigate the problem of string-based molecular generation via variational autoencoders (VAEs) We propose a simple, yet effective idea to improve the performance of VAE for the task. In our experiments, the proposed VAE model particularly performs well for generating a sample from out-of-domain distribution.
  arXiv  Detail & Related papers  (2022-08-23T03:56:30Z)
• Consistency Regularization for Variational Auto-Encoders [14.423556966548544]
  Variational auto-encoders (VAEs) are a powerful approach to unsupervised learning. We propose a regularization method to enforce consistency in VAEs.
  arXiv  Detail & Related papers  (2021-05-31T10:26:32Z)
• Improved Training of Sparse Coding Variational Autoencoder via Weight Normalization [0.0]
  We focus on a recently proposed model, sparse coding variational autoencoder (SVAE) We show that projection of the filters onto unit norm drastically increases the number of active filters. Our results highlight the importance of weight normalization for learning sparse representation from data.
  arXiv  Detail & Related papers  (2021-01-23T08:07:20Z)
• Autoencoding Variational Autoencoder [56.05008520271406]
  We study the implications of this behaviour on the learned representations and also the consequences of fixing it by introducing a notion of self consistency. We show that encoders trained with our self-consistency approach lead to representations that are robust (insensitive) to perturbations in the input introduced by adversarial attacks.
  arXiv  Detail & Related papers  (2020-12-07T14:16:14Z)
• A New Modal Autoencoder for Functionally Independent Feature Extraction [6.690183908967779]
  A new modal autoencoder (MAE) is proposed by othogonalising the columns of the readout weight matrix. The results were validated on the MNIST variations and USPS classification benchmark suite. The new MAE introduces a very simple training principle for autoencoders and could be promising for the pre-training of deep neural networks.
  arXiv  Detail & Related papers  (2020-06-25T13:25:10Z)
• Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shift [81.74795324629712]
  We call prediction-time batch normalization, which significantly improves model accuracy and calibration under covariate shift. We show that prediction-time batch normalization provides complementary benefits to existing state-of-the-art approaches for improving robustness. The method has mixed results when used alongside pre-training, and does not seem to perform as well under more natural types of dataset shift.
  arXiv  Detail & Related papers  (2020-06-19T05:08:43Z)
• Dynamic Scale Training for Object Detection [111.33112051962514]
  We propose a Dynamic Scale Training paradigm (abbreviated as DST) to mitigate scale variation challenge in object detection. Experimental results demonstrate the efficacy of our proposed DST towards scale variation handling. It does not introduce inference overhead and could serve as a free lunch for general detection configurations.
  arXiv  Detail & Related papers  (2020-04-26T16:48:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.