Understanding Variational Inference in Function-Space

• URL: http://arxiv.org/abs/2011.09421v1
• Date: Wed, 18 Nov 2020 17:42:01 GMT
• Title: Understanding Variational Inference in Function-Space
• Authors: David R. Burt, Sebastian W. Ober, Adri\`a Garriga-Alonso, Mark van der Wilk
• Abstract summary: We highlight some advantages and limitations of employing the Kullback-Leibler divergence in this setting. We propose (featurized) Bayesian linear regression as a benchmark for function-space' inference methods that directly measures approximation quality.
• Score: 20.940162027560408
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent work has attempted to directly approximate the `function-space' or predictive posterior distribution of Bayesian models, without approximating the posterior distribution over the parameters. This is appealing in e.g. Bayesian neural networks, where we only need the former, and the latter is hard to represent. In this work, we highlight some advantages and limitations of employing the Kullback-Leibler divergence in this setting. For example, we show that minimizing the KL divergence between a wide class of parametric distributions and the posterior induced by a (non-degenerate) Gaussian process prior leads to an ill-defined objective function. Then, we propose (featurized) Bayesian linear regression as a benchmark for `function-space' inference methods that directly measures approximation quality. We apply this methodology to assess aspects of the objective function and inference scheme considered in Sun, Zhang, Shi, and Grosse (2018), emphasizing the quality of approximation to Bayesian inference as opposed to predictive performance.

Related papers

• von Mises Quasi-Processes for Bayesian Circular Regression [57.88921637944379]
  We explore a family of expressive and interpretable distributions over circle-valued random functions. The resulting probability model has connections with continuous spin models in statistical physics. For posterior inference, we introduce a new Stratonovich-like augmentation that lends itself to fast Markov Chain Monte Carlo sampling.
  arXiv  Detail & Related papers  (2024-06-19T01:57:21Z)
• Reparameterization invariance in approximate Bayesian inference [32.88960624085645]
  We develop a new geometric view of reparametrizations from which we explain the success of linearization. We demonstrate that these re parameterization invariance properties can be extended to the original neural network predictive.
  arXiv  Detail & Related papers  (2024-06-05T14:49:15Z)
• Implicit Variational Inference for High-Dimensional Posteriors [7.924706533725115]
  In variational inference, the benefits of Bayesian models rely on accurately capturing the true posterior distribution. We propose using neural samplers that specify implicit distributions, which are well-suited for approximating complex multimodal and correlated posteriors. Our approach introduces novel bounds for approximate inference using implicit distributions by locally linearising the neural sampler.
  arXiv  Detail & Related papers  (2023-10-10T14:06:56Z)
• Promises and Pitfalls of the Linearized Laplace in Bayesian Optimization [73.80101701431103]
  The linearized-Laplace approximation (LLA) has been shown to be effective and efficient in constructing Bayesian neural networks. We study the usefulness of the LLA in Bayesian optimization and highlight its strong performance and flexibility.
  arXiv  Detail & Related papers  (2023-04-17T14:23:43Z)
• Kernel-based off-policy estimation without overlap: Instance optimality beyond semiparametric efficiency [53.90687548731265]
  We study optimal procedures for estimating a linear functional based on observational data. For any convex and symmetric function class $mathcalF$, we derive a non-asymptotic local minimax bound on the mean-squared error.
  arXiv  Detail & Related papers  (2023-01-16T02:57:37Z)
• On the detrimental effect of invariances in the likelihood for variational inference [21.912271882110986]
  Variational Bayesian posterior inference often requires simplifying approximations such as mean-field parametrisation to ensure tractability. Prior work has associated the variational mean-field approximation for Bayesian neural networks with underfitting in the case of small datasets or large model sizes.
  arXiv  Detail & Related papers  (2022-09-15T09:13:30Z)
• Loss-calibrated expectation propagation for approximate Bayesian decision-making [24.975981795360845]
  We introduce loss-calibrated expectation propagation (Loss-EP), a loss-calibrated variant of expectation propagation. We show how this asymmetry can have dramatic consequences on what information is "useful" to capture in an approximation.
  arXiv  Detail & Related papers  (2022-01-10T01:42:28Z)
• Variational Refinement for Importance Sampling Using the Forward Kullback-Leibler Divergence [77.06203118175335]
  Variational Inference (VI) is a popular alternative to exact sampling in Bayesian inference. Importance sampling (IS) is often used to fine-tune and de-bias the estimates of approximate Bayesian inference procedures. We propose a novel combination of optimization and sampling techniques for approximate Bayesian inference.
  arXiv  Detail & Related papers  (2021-06-30T11:00:24Z)
• Variational Inference as Iterative Projection in a Bayesian Hilbert Space with Application to Robotic State Estimation [14.670851095242451]
  Variational Bayesian inference is an important machine-learning tool that finds application from statistics to robotics. We show that variational inference based on KL divergence can amount to iterative projection, in the Euclidean sense, of the Bayesian posterior onto a subspace corresponding to the selected approximation family.
  arXiv  Detail & Related papers  (2020-05-14T21:33:31Z)
• Bayesian Deep Learning and a Probabilistic Perspective of Generalization [56.69671152009899]
  We show that deep ensembles provide an effective mechanism for approximate Bayesian marginalization. We also propose a related approach that further improves the predictive distribution by marginalizing within basins of attraction.
  arXiv  Detail & Related papers  (2020-02-20T15:13:27Z)
• The k-tied Normal Distribution: A Compact Parameterization of Gaussian Mean Field Posteriors in Bayesian Neural Networks [46.677567663908185]
  Variational Bayesian Inference is a popular methodology for approxing posteriorimating over Bayesian neural network weights. Recent work has explored ever richer parameterizations of the approximate posterior in the hope of improving performance. We find that by decomposing these variational parameters into a low-rank factorization, we can make our variational approximation more compact without decreasing the models' performance.
  arXiv  Detail & Related papers  (2020-02-07T07:33:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.