Challenges and Opportunities in High-dimensional Variational Inference

• URL: http://arxiv.org/abs/2103.01085v1
• Date: Mon, 1 Mar 2021 15:53:34 GMT
• Title: Challenges and Opportunities in High-dimensional Variational Inference
• Authors: Akash Kumar Dhaka, Alejandro Catalina, Manushi Welandawe, Michael Riis Andersen, Jonathan Huggins, Aki Vehtari
• Abstract summary: We show why intuitions about approximate families and divergences for low-dimensional posteriors fail for higher-dimensional posteriors. For high-dimensional posteriors we recommend using the exclusive KL divergence that is most stable and easiest to optimize. In low to moderate dimensions, heavy-tailed variational families and mass-covering divergences can increase the chances that the approximation can be improved by importance sampling.
• Score: 65.53746326245059
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We explore the limitations of and best practices for using black-box variational inference to estimate posterior summaries of the model parameters. By taking an importance sampling perspective, we are able to explain and empirically demonstrate: 1) why the intuitions about the behavior of approximate families and divergences for low-dimensional posteriors fail for higher-dimensional posteriors, 2) how we can diagnose the pre-asymptotic reliability of variational inference in practice by examining the behavior of the density ratios (i.e., importance weights), 3) why the choice of variational objective is not as relevant for higher-dimensional posteriors, and 4) why, although flexible variational families can provide some benefits in higher dimensions, they also introduce additional optimization challenges. Based on these findings, for high-dimensional posteriors we recommend using the exclusive KL divergence that is most stable and easiest to optimize, and then focusing on improving the variational family or using model parameter transformations to make the posterior more similar to the approximating family. Our results also show that in low to moderate dimensions, heavy-tailed variational families and mass-covering divergences can increase the chances that the approximation can be improved by importance sampling.

Related papers

• Towards Improved Variational Inference for Deep Bayesian Models [7.841254447222393]
  In this thesis, we explore the use of variational inference (VI) as an approximation. VI is unique in simultaneously approximating the posterior and providing a lower bound to the marginal likelihood. We propose a variational posterior that provides a unified view of inference in Bayesian neural networks and deep Gaussian processes.
  arXiv  Detail & Related papers  (2024-01-23T00:40:20Z)
• Forward $\chi^2$ Divergence Based Variational Importance Sampling [2.841087763205822]
  We introduce a novel variational importance sampling (VIS) approach that directly estimates and maximizes the log-likelihood. We apply VIS to various popular latent variable models, including mixture models, variational auto-encoders, and partially observable generalized linear models.
  arXiv  Detail & Related papers  (2023-11-04T21:46:28Z)
• Variational autoencoder with weighted samples for high-dimensional non-parametric adaptive importance sampling [0.0]
  We extend the existing framework to the case of weighted samples by introducing a new objective function. In order to add flexibility to the model and to be able to learn multimodal distributions, we consider a learnable prior distribution. We exploit the proposed procedure in existing adaptive importance sampling algorithms to draw points from a target distribution and to estimate a rare event probability in high dimension.
  arXiv  Detail & Related papers  (2023-10-13T15:40:55Z)
• High Dimensional Causal Inference with Variational Backdoor Adjustment [57.31312942774617]
  We take a generative modeling approach to backdoor adjustment for high dimensional treatments and confounders. Our method is able to estimate interventional likelihood in a variety of high dimensional settings, including semi-synthetic X-ray medical data.
  arXiv  Detail & Related papers  (2023-10-09T19:21:41Z)
• Reparameterized Variational Rejection Sampling [12.189621777178354]
  Variational Rejection Sampling (VRS) combines a parametric proposal distribution with sampling rejection to define a rich non-parametric family of distributions. We show that our method performs well in practice and that it is well-suited for black-box inference, especially for models with local latent variables.
  arXiv  Detail & Related papers  (2023-09-26T01:46:53Z)
• A Tutorial on Parametric Variational Inference [0.0]
  Variational inference is now the preferred choice for many high-dimensional models and large datasets. This tutorial introduces variational inference from the parametric perspective that dominates these recent developments.
  arXiv  Detail & Related papers  (2023-01-03T17:30:50Z)
• Monotonicity and Double Descent in Uncertainty Estimation with Gaussian Processes [52.92110730286403]
  It is commonly believed that the marginal likelihood should be reminiscent of cross-validation metrics and that both should deteriorate with larger input dimensions. We prove that by tuning hyper parameters, the performance, as measured by the marginal likelihood, improves monotonically with the input dimension. We also prove that cross-validation metrics exhibit qualitatively different behavior that is characteristic of double descent.
  arXiv  Detail & Related papers  (2022-10-14T08:09:33Z)
• Evaluating Sensitivity to the Stick-Breaking Prior in Bayesian Nonparametrics [85.31247588089686]
  We show that variational Bayesian methods can yield sensitivities with respect to parametric and nonparametric aspects of Bayesian models. We provide both theoretical and empirical support for our variational approach to Bayesian sensitivity analysis.
  arXiv  Detail & Related papers  (2021-07-08T03:40:18Z)
• Variational Causal Networks: Approximate Bayesian Inference over Causal Structures [132.74509389517203]
  We introduce a parametric variational family modelled by an autoregressive distribution over the space of discrete DAGs. In experiments, we demonstrate that the proposed variational posterior is able to provide a good approximation of the true posterior.
  arXiv  Detail & Related papers  (2021-06-14T17:52:49Z)
• Robust, Accurate Stochastic Optimization for Variational Inference [68.83746081733464]
  We show that common optimization methods lead to poor variational approximations if the problem is moderately large. Motivated by these findings, we develop a more robust and accurate optimization framework by viewing the underlying algorithm as producing a Markov chain.
  arXiv  Detail & Related papers  (2020-09-01T19:12:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.