ELBOing Stein: Variational Bayes with Stein Mixture Inference

• URL: http://arxiv.org/abs/2410.22948v1
• Date: Wed, 30 Oct 2024 12:05:12 GMT
• Title: ELBOing Stein: Variational Bayes with Stein Mixture Inference
• Authors: Ola Rønning, Eric Nalisnick, Christophe Ley, Padhraic Smyth, Thomas Hamelryck,
• Abstract summary: Stein variational descent (SVGD) performs approximate Bayesian inference by representing the posterior with a set of particles. We generalize SVGD by letting each particle parameterize a component distribution in a mixture model. Our method, Stein Mixture Inference (SMI), optimize a lower bound to the evidence (ELBO) and introduces user-specified guides.
• Score: 12.562946804046051
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Stein variational gradient descent (SVGD) [Liu and Wang, 2016] performs approximate Bayesian inference by representing the posterior with a set of particles. However, SVGD suffers from variance collapse, i.e. poor predictions due to underestimating uncertainty [Ba et al., 2021], even for moderately-dimensional models such as small Bayesian neural networks (BNNs). To address this issue, we generalize SVGD by letting each particle parameterize a component distribution in a mixture model. Our method, Stein Mixture Inference (SMI), optimizes a lower bound to the evidence (ELBO) and introduces user-specified guides parameterized by particles. SMI extends the Nonlinear SVGD framework [Wang and Liu, 2019] to the case of variational Bayes. SMI effectively avoids variance collapse, judging by a previously described test developed for this purpose, and performs well on standard data sets. In addition, SMI requires considerably fewer particles than SVGD to accurately estimate uncertainty for small BNNs. The synergistic combination of NSVGD, ELBO optimization and user-specified guides establishes a promising approach towards variational Bayesian inference in the case of tall and wide data.

Related papers

• Rejection via Learning Density Ratios [50.91522897152437]
  Classification with rejection emerges as a learning paradigm which allows models to abstain from making predictions.<n>We propose a different distributional perspective, where we seek to find an idealized data distribution which maximizes a pretrained model's performance.<n>Our framework is tested empirically over clean and noisy datasets.
  arXiv  Detail & Related papers  (2024-05-29T01:32:17Z)
• Augmented Message Passing Stein Variational Gradient Descent [3.5788754401889014]
  We study the isotropy property of finite particles during the convergence process. All particles tend to cluster around the particle center within a certain range. Our algorithm achieves satisfactory accuracy and overcomes the variance collapse problem in various benchmark problems.
  arXiv  Detail & Related papers  (2023-05-18T01:13:04Z)
• Grassmann Stein Variational Gradient Descent [3.644031721554146]
  Stein variational gradient descent (SVGD) is a deterministic particle inference algorithm that provides an efficient alternative to Markov chain Monte Carlo. Recent developments have advocated projecting both the score function and the data onto real lines to sidestep this issue. We propose Grassmann Stein variational gradient descent (GSVGD) as an alternative approach, which permits projections onto arbitrary dimensional subspaces.
  arXiv  Detail & Related papers  (2022-02-07T15:36:03Z)
• Scaling Structured Inference with Randomization [64.18063627155128]
  We propose a family of dynamic programming (RDP) randomized for scaling structured models to tens of thousands of latent states. Our method is widely applicable to classical DP-based inference. It is also compatible with automatic differentiation so can be integrated with neural networks seamlessly.
  arXiv  Detail & Related papers  (2021-12-07T11:26:41Z)
• Forget-SVGD: Particle-Based Bayesian Federated Unlearning [32.638916321653554]
  Forget-Stein Variational Gradient Descent (Forget-SVGD) builds on SVGD. The proposed method is validated via performance comparisons with non-parametric schemes that train from scratch by excluding data to be forgotten.
  arXiv  Detail & Related papers  (2021-11-23T18:15:50Z)
• Inverting brain grey matter models with likelihood-free inference: a tool for trustable cytoarchitecture measurements [62.997667081978825]
  characterisation of the brain grey matter cytoarchitecture with quantitative sensitivity to soma density and volume remains an unsolved challenge in dMRI. We propose a new forward model, specifically a new system of equations, requiring a few relatively sparse b-shells. We then apply modern tools from Bayesian analysis known as likelihood-free inference (LFI) to invert our proposed model.
  arXiv  Detail & Related papers  (2021-11-15T09:08:27Z)
• Post-mortem on a deep learning contest: a Simpson's paradox and the complementary roles of scale metrics versus shape metrics [61.49826776409194]
  We analyze a corpus of models made publicly-available for a contest to predict the generalization accuracy of neural network (NN) models. We identify what amounts to a Simpson's paradox: where "scale" metrics perform well overall but perform poorly on sub partitions of the data. We present two novel shape metrics, one data-independent, and the other data-dependent, which can predict trends in the test accuracy of a series of NNs.
  arXiv  Detail & Related papers  (2021-06-01T19:19:49Z)
• Sampling-free Variational Inference for Neural Networks with Multiplicative Activation Noise [51.080620762639434]
  We propose a more efficient parameterization of the posterior approximation for sampling-free variational inference. Our approach yields competitive results for standard regression problems and scales well to large-scale image classification tasks.
  arXiv  Detail & Related papers  (2021-03-15T16:16:18Z)
• Federated Generalized Bayesian Learning via Distributed Stein Variational Gradient Descent [38.41707037232561]
  This paper introduces Distributed Stein Variational Gradient Descent (DSVGD), a non-parametric generalized Bayesian inference framework for federated learning. By varying the number of particles, DSVGD enables a flexible trade-off between per-iteration communication load and number of communication rounds.
  arXiv  Detail & Related papers  (2020-09-11T17:33:22Z)
• Kernel Stein Generative Modeling [68.03537693810972]
  Gradient Langevin Dynamics (SGLD) demonstrates impressive results with energy-based models on high-dimensional and complex data distributions. Stein Variational Gradient Descent (SVGD) is a deterministic sampling algorithm that iteratively transports a set of particles to approximate a given distribution. We propose noise conditional kernel SVGD (NCK-SVGD), that works in tandem with the recently introduced Noise Conditional Score Network estimator.
  arXiv  Detail & Related papers  (2020-07-06T21:26:04Z)
• Sliced Kernelized Stein Discrepancy [17.159499204595527]
  Kernelized Stein discrepancy (KSD) is extensively used in goodness-of-fit tests and model learning. We propose the sliced Stein discrepancy and its scalable and kernelized variants, which employ kernel-based test functions defined on the optimal one-dimensional projections. For model learning, we show its advantages over existing Stein discrepancy baselines by training independent component analysis models with different discrepancies.
  arXiv  Detail & Related papers  (2020-06-30T04:58:55Z)
• Stein Variational Inference for Discrete Distributions [70.19352762933259]
  We propose a simple yet general framework that transforms discrete distributions to equivalent piecewise continuous distributions. Our method outperforms traditional algorithms such as Gibbs sampling and discontinuous Hamiltonian Monte Carlo. We demonstrate that our method provides a promising tool for learning ensembles of binarized neural network (BNN) In addition, such transform can be straightforwardly employed in gradient-free kernelized Stein discrepancy to perform goodness-of-fit (GOF) test on discrete distributions.
  arXiv  Detail & Related papers  (2020-03-01T22:45:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.