Automatic Differentiation Variational Inference with Mixtures

• URL: http://arxiv.org/abs/2003.01687v4
• Date: Wed, 24 Jun 2020 17:35:38 GMT
• Title: Automatic Differentiation Variational Inference with Mixtures
• Authors: Warren R. Morningstar, Sharad M. Vikram, Cusuh Ham, Andrew Gallagher, Joshua V. Dillon
• Abstract summary: We show how stratified sampling may be used to enable mixture distributions as the approximate posterior. We derive a new lower bound on the evidence analogous to the importance weighted autoencoder (IWAE)
• Score: 4.995383193706478
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Automatic Differentiation Variational Inference (ADVI) is a useful tool for efficiently learning probabilistic models in machine learning. Generally approximate posteriors learned by ADVI are forced to be unimodal in order to facilitate use of the reparameterization trick. In this paper, we show how stratified sampling may be used to enable mixture distributions as the approximate posterior, and derive a new lower bound on the evidence analogous to the importance weighted autoencoder (IWAE). We show that this "SIWAE" is a tighter bound than both IWAE and the traditional ELBO, both of which are special instances of this bound. We verify empirically that the traditional ELBO objective disfavors the presence of multimodal posterior distributions and may therefore not be able to fully capture structure in the latent space. Our experiments show that using the SIWAE objective allows the encoder to learn more complex distributions which regularly contain multimodality, resulting in higher accuracy and better calibration in the presence of incomplete, limited, or corrupted data.

Related papers

• FFHFlow: A Flow-based Variational Approach for Multi-fingered Grasp Synthesis in Real Time [19.308304984645684]
  We propose exploiting a special kind of Deep Generative Model (DGM) based on Normalizing Flows (NFs) We first observed an encouraging improvement in diversity by directly applying a single conditional NFs (cNFs) to learn a grasp distribution conditioned on the incomplete point cloud. This motivated us to develop a novel flow-based d Deep Latent Variable Model (DLVM) Unlike Variational Autoencoders (VAEs), the proposed DLVM counteracts typical pitfalls by leveraging two cNFs for the prior and likelihood distributions.
  arXiv  Detail & Related papers  (2024-07-21T13:33:08Z)
• Mitigating Shortcut Learning with Diffusion Counterfactuals and Diverse Ensembles [95.49699178874683]
  We propose DiffDiv, an ensemble diversification framework exploiting Diffusion Probabilistic Models (DPMs) We show that DPMs can generate images with novel feature combinations, even when trained on samples displaying correlated input features. We show that DPM-guided diversification is sufficient to remove dependence on shortcut cues, without a need for additional supervised signals.
  arXiv  Detail & Related papers  (2023-11-23T15:47:33Z)
• Leveraging Diffusion Disentangled Representations to Mitigate Shortcuts in Underspecified Visual Tasks [92.32670915472099]
  We propose an ensemble diversification framework exploiting the generation of synthetic counterfactuals using Diffusion Probabilistic Models (DPMs) We show that diffusion-guided diversification can lead models to avert attention from shortcut cues, achieving ensemble diversity performance comparable to previous methods requiring additional data collection.
  arXiv  Detail & Related papers  (2023-10-03T17:37:52Z)
• Symmetric Equilibrium Learning of VAEs [56.56929742714685]
  We view variational autoencoders (VAEs) as decoder-encoder pairs, which map distributions in the data space to distributions in the latent space and vice versa. We propose a Nash equilibrium learning approach, which is symmetric with respect to the encoder and decoder and allows learning VAEs in situations where both the data and the latent distributions are accessible only by sampling.
  arXiv  Detail & Related papers  (2023-07-19T10:27:34Z)
• RegMixup: Mixup as a Regularizer Can Surprisingly Improve Accuracy and Out Distribution Robustness [94.69774317059122]
  We show that the effectiveness of the well celebrated Mixup can be further improved if instead of using it as the sole learning objective, it is utilized as an additional regularizer to the standard cross-entropy loss. This simple change not only provides much improved accuracy but also significantly improves the quality of the predictive uncertainty estimation of Mixup.
  arXiv  Detail & Related papers  (2022-06-29T09:44:33Z)
• Regularizing Variational Autoencoder with Diversity and Uncertainty Awareness [61.827054365139645]
  Variational Autoencoder (VAE) approximates the posterior of latent variables based on amortized variational inference. We propose an alternative model, DU-VAE, for learning a more Diverse and less Uncertain latent space.
  arXiv  Detail & Related papers  (2021-10-24T07:58:13Z)
• InteL-VAEs: Adding Inductive Biases to Variational Auto-Encoders via Intermediary Latents [60.785317191131284]
  We introduce a simple and effective method for learning VAEs with controllable biases by using an intermediary set of latent variables. In particular, it allows us to impose desired properties like sparsity or clustering on learned representations. We show that this, in turn, allows InteL-VAEs to learn both better generative models and representations.
  arXiv  Detail & Related papers  (2021-06-25T16:34:05Z)
• To Regularize or Not To Regularize? The Bias Variance Trade-off in Regularized AEs [10.611727286504994]
  We study the effect of the latent prior on the generation deterministic quality of AE models. We show that our model, called FlexAE, is the new state-of-the-art for the AE based generative models.
  arXiv  Detail & Related papers  (2020-06-10T14:00:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.