Bayesian Online Natural Gradient (BONG)

• URL: http://arxiv.org/abs/2405.19681v2
• Date: Thu, 31 Oct 2024 07:25:58 GMT
• Title: Bayesian Online Natural Gradient (BONG)
• Authors: Matt Jones, Peter Chang, Kevin Murphy,
• Abstract summary: We propose a novel approach to sequential Bayesian inference based on variational Bayes (VB) The key insight is that, in the online setting, we do not need to add the KL term to regularize to the prior. We show empirically that our method outperforms other online VB methods in the non-conjugate setting.
• Score: 9.800443064368467
• License:
• Abstract: We propose a novel approach to sequential Bayesian inference based on variational Bayes (VB). The key insight is that, in the online setting, we do not need to add the KL term to regularize to the prior (which comes from the posterior at the previous timestep); instead we can optimize just the expected log-likelihood, performing a single step of natural gradient descent starting at the prior predictive. We prove this method recovers exact Bayesian inference if the model is conjugate. We also show how to compute an efficient deterministic approximation to the VB objective, as well as our simplified objective, when the variational distribution is Gaussian or a sub-family, including the case of a diagonal plus low-rank precision matrix. We show empirically that our method outperforms other online VB methods in the non-conjugate setting, such as online learning for neural networks, especially when controlling for computational costs.

Related papers

• Calibrating Neural Simulation-Based Inference with Differentiable Coverage Probability [50.44439018155837]
  We propose to include a calibration term directly into the training objective of the neural model. By introducing a relaxation of the classical formulation of calibration error we enable end-to-end backpropagation. It is directly applicable to existing computational pipelines allowing reliable black-box posterior inference.
  arXiv  Detail & Related papers  (2023-10-20T10:20:45Z)
• Low-rank extended Kalman filtering for online learning of neural networks from streaming data [71.97861600347959]
  We propose an efficient online approximate Bayesian inference algorithm for estimating the parameters of a nonlinear function from a potentially non-stationary data stream. The method is based on the extended Kalman filter (EKF), but uses a novel low-rank plus diagonal decomposition of the posterior matrix. In contrast to methods based on variational inference, our method is fully deterministic, and does not require step-size tuning.
  arXiv  Detail & Related papers  (2023-05-31T03:48:49Z)
• Variational Linearized Laplace Approximation for Bayesian Deep Learning [11.22428369342346]
  We propose a new method for approximating Linearized Laplace Approximation (LLA) using a variational sparse Gaussian Process (GP) Our method is based on the dual RKHS formulation of GPs and retains, as the predictive mean, the output of the original DNN. It allows for efficient optimization, which results in sub-linear training time in the size of the training dataset.
  arXiv  Detail & Related papers  (2023-02-24T10:32:30Z)
• Sparse high-dimensional linear regression with a partitioned empirical Bayes ECM algorithm [62.997667081978825]
  We propose a computationally efficient and powerful Bayesian approach for sparse high-dimensional linear regression. Minimal prior assumptions on the parameters are used through the use of plug-in empirical Bayes estimates. The proposed approach is implemented in the R package probe.
  arXiv  Detail & Related papers  (2022-09-16T19:15:50Z)
• Using Random Walks for Iterative Phase Estimation [12.892284518456059]
  We provide a new approach to online Bayesian phase estimation that achieves Heisenberg limited scaling. This practically means that we can perform an update in microseconds on a CPU as opposed to milliseconds for existing particle filter methods. This work shows that online Bayesian inference is practical, efficient and ready for deployment in modern FPGA driven adaptive experiments.
  arXiv  Detail & Related papers  (2022-08-09T03:31:15Z)
• Quasi Black-Box Variational Inference with Natural Gradients for Bayesian Learning [84.90242084523565]
  We develop an optimization algorithm suitable for Bayesian learning in complex models. Our approach relies on natural gradient updates within a general black-box framework for efficient training with limited model-specific derivations.
  arXiv  Detail & Related papers  (2022-05-23T18:54:27Z)
• Scalable Marginal Likelihood Estimation for Model Selection in Deep Learning [78.83598532168256]
  Marginal-likelihood based model-selection is rarely used in deep learning due to estimation difficulties. Our work shows that marginal likelihoods can improve generalization and be useful when validation data is unavailable.
  arXiv  Detail & Related papers  (2021-04-11T09:50:24Z)
• Improving predictions of Bayesian neural nets via local linearization [79.21517734364093]
  We argue that the Gauss-Newton approximation should be understood as a local linearization of the underlying Bayesian neural network (BNN) Because we use this linearized model for posterior inference, we should also predict using this modified model instead of the original one. We refer to this modified predictive as "GLM predictive" and show that it effectively resolves common underfitting problems of the Laplace approximation.
  arXiv  Detail & Related papers  (2020-08-19T12:35:55Z)
• The FMRIB Variational Bayesian Inference Tutorial II: Stochastic Variational Bayes [1.827510863075184]
  This tutorial revisits the original FMRIB Variational Bayes tutorial. This new approach bears a lot of similarity to, and has benefited from, computational methods applied to machine learning algorithms.
  arXiv  Detail & Related papers  (2020-07-03T11:31:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.