Statistical Foundation of Variational Bayes Neural Networks

• URL: http://arxiv.org/abs/2006.15786v1
• Date: Mon, 29 Jun 2020 03:04:18 GMT
• Title: Statistical Foundation of Variational Bayes Neural Networks
• Authors: Shrijita Bhattacharya and Tapabrata Maiti
• Abstract summary: Variational Bayes (VB) provides a useful alternative to circumvent the computational cost and time complexity associated with the generation of samples from the true posterior. This paper establishes the fundamental result of posterior consistency for the mean-field variational posterior (VP) for a feed-forward artificial neural network model.
• Score: 0.456877715768796
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Despite the popularism of Bayesian neural networks in recent years, its use is somewhat limited in complex and big data situations due to the computational cost associated with full posterior evaluations. Variational Bayes (VB) provides a useful alternative to circumvent the computational cost and time complexity associated with the generation of samples from the true posterior using Markov Chain Monte Carlo (MCMC) techniques. The efficacy of the VB methods is well established in machine learning literature. However, its potential broader impact is hindered due to a lack of theoretical validity from a statistical perspective. However there are few results which revolve around the theoretical properties of VB, especially in non-parametric problems. In this paper, we establish the fundamental result of posterior consistency for the mean-field variational posterior (VP) for a feed-forward artificial neural network model. The paper underlines the conditions needed to guarantee that the VP concentrates around Hellinger neighborhoods of the true density function. Additionally, the role of the scale parameter and its influence on the convergence rates has also been discussed. The paper mainly relies on two results (1) the rate at which the true posterior grows (2) the rate at which the KL-distance between the posterior and variational posterior grows. The theory provides a guideline of building prior distributions for Bayesian NN models along with an assessment of accuracy of the corresponding VB implementation.

Related papers

• Uncertainty Prediction Neural Network (UpNet): Embedding Artificial Neural Network in Bayesian Inversion Framework to Quantify the Uncertainty of Remote Sensing Retrieval [0.34952465649465553]
  Inversion of radiative transfer models (RTMs) is the most commonly used approach to retrieve large-scale vegetation biophysical parameters. In recent years, Artificial Neural Network (ANN)-based methods have become the mainstream for inverting RTMs due to their high accuracy and computational efficiency. Due to the lack of the Bayesian inversion theory interpretation, it faces challenges in quantifying the retrieval uncertainty.
  arXiv  Detail & Related papers  (2024-11-07T09:27:42Z)
• Posterior and variational inference for deep neural networks with heavy-tailed weights [0.0]
  We consider deep neural networks in a Bayesian framework with a prior distribution sampling the network weights at random. We show that the corresponding posterior distribution achieves near-optimal minimax contraction rates. We also provide variational Bayes counterparts of the results, that show that mean-field variational approximations still benefit from near-optimal theoretical support.
  arXiv  Detail & Related papers  (2024-06-05T15:24:20Z)
• Evidence Networks: simple losses for fast, amortized, neural Bayesian model comparison [0.0]
  Evidence Networks can enable Bayesian model comparison when state-of-the-art methods fail. We introduce the leaky parity-odd power transform, leading to the novel l-POP-Exponential'' loss function. We show that Evidence Networks are explicitly independent of dimensionality of the parameter space and scale mildly with the complexity of the posterior probability density function.
  arXiv  Detail & Related papers  (2023-05-18T18:14:53Z)
• Neural Importance Sampling for Rapid and Reliable Gravitational-Wave Inference [59.040209568168436]
  We first generate a rapid proposal for the Bayesian posterior using neural networks, and then attach importance weights based on the underlying likelihood and prior. This provides (1) a corrected posterior free from network inaccuracies, (2) a performance diagnostic (the sample efficiency) for assessing the proposal and identifying failure cases, and (3) an unbiased estimate of the Bayesian evidence. We carry out a large study analyzing 42 binary black hole mergers observed by LIGO and Virgo with the SEOBNRv4PHM and IMRPhenomHMXP waveform models.
  arXiv  Detail & Related papers  (2022-10-11T18:00:02Z)
• How Tempering Fixes Data Augmentation in Bayesian Neural Networks [22.188535244056016]
  We show that tempering implicitly reduces the misspecification arising from modeling augmentations as i.i.d. data. The temperature mimics the role of the effective sample size, reflecting the gain in information provided by the augmentations.
  arXiv  Detail & Related papers  (2022-05-27T11:06:56Z)
• Test-time Batch Statistics Calibration for Covariate Shift [66.7044675981449]
  We propose to adapt the deep models to the novel environment during inference. We present a general formulation $alpha$-BN to calibrate the batch statistics. We also present a novel loss function to form a unified test time adaptation framework Core.
  arXiv  Detail & Related papers  (2021-10-06T08:45:03Z)
• An Uncertainty-Driven GCN Refinement Strategy for Organ Segmentation [53.425900196763756]
  We propose a segmentation refinement method based on uncertainty analysis and graph convolutional networks. We employ the uncertainty levels of the convolutional network in a particular input volume to formulate a semi-supervised graph learning problem. We show that our method outperforms the state-of-the-art CRF refinement method by improving the dice score by 1% for the pancreas and 2% for spleen.
  arXiv  Detail & Related papers  (2020-12-06T18:55:07Z)
• Variational Bayes Neural Network: Posterior Consistency, Classification Accuracy and Computational Challenges [0.3867363075280544]
  This paper develops a variational Bayesian neural network estimation methodology and related statistical theory. The development is motivated by an important biomedical engineering application, namely building predictive tools for the transition from mild cognitive impairment to Alzheimer's disease.
  arXiv  Detail & Related papers  (2020-11-19T00:11:27Z)
• Unlabelled Data Improves Bayesian Uncertainty Calibration under Covariate Shift [100.52588638477862]
  We develop an approximate Bayesian inference scheme based on posterior regularisation. We demonstrate the utility of our method in the context of transferring prognostic models of prostate cancer across globally diverse populations.
  arXiv  Detail & Related papers  (2020-06-26T13:50:19Z)
• Revisiting Initialization of Neural Networks [72.24615341588846]
  We propose a rigorous estimation of the global curvature of weights across layers by approximating and controlling the norm of their Hessian matrix. Our experiments on Word2Vec and the MNIST/CIFAR image classification tasks confirm that tracking the Hessian norm is a useful diagnostic tool.
  arXiv  Detail & Related papers  (2020-04-20T18:12:56Z)
• Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks [65.24701908364383]
  We show that a sufficient condition for a uncertainty on a ReLU network is "to be a bit Bayesian calibrated" We further validate these findings empirically via various standard experiments using common deep ReLU networks and Laplace approximations.
  arXiv  Detail & Related papers  (2020-02-24T08:52:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.