Function-Space MCMC for Bayesian Wide Neural Networks

• URL: http://arxiv.org/abs/2408.14325v3
• Date: Mon, 7 Oct 2024 14:23:50 GMT
• Title: Function-Space MCMC for Bayesian Wide Neural Networks
• Authors: Lucia Pezzetti, Stefano Favaro, Stefano Peluchetti,
• Abstract summary: We investigate the use of the preconditioned Crank-Nicolson algorithm and its Langevin version to sample from the reparametrised posterior distribution of the weights. We prove that the acceptance probabilities of the proposed methods approach 1 as the width of the network increases.
• Score: 9.899763598214124
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Bayesian Neural Networks represent a fascinating confluence of deep learning and probabilistic reasoning, offering a compelling framework for understanding uncertainty in complex predictive models. In this paper, we investigate the use of the preconditioned Crank-Nicolson algorithm and its Langevin version to sample from the reparametrised posterior distribution of the weights as the widths of Bayesian Neural Networks grow larger. In addition to being robust in the infinite-dimensional setting, we prove that the acceptance probabilities of the proposed methods approach 1 as the width of the network increases, independently of any stepsize tuning. Moreover, we examine and compare how the mixing speeds of the underdamped Langevin Monte Carlo, the preconditioned Crank-Nicolson and the preconditioned Crank-Nicolson Langevin samplers are influenced by changes in the network width in some real-world cases. Our findings suggest that, in wide Bayesian Neural Networks configurations, the preconditioned Crank-Nicolson method allows for more efficient sampling of the reparametrised posterior distribution, as evidenced by a higher effective sample size and improved diagnostic results compared with the other analysed algorithms.

Related papers

• Posterior and variational inference for deep neural networks with heavy-tailed weights [0.0]
  We consider deep neural networks in a Bayesian framework with a prior distribution sampling the network weights at random. We show that the corresponding posterior distribution achieves near-optimal minimax contraction rates. We also provide variational Bayes counterparts of the results, that show that mean-field variational approximations still benefit from near-optimal theoretical support.
  arXiv  Detail & Related papers  (2024-06-05T15:24:20Z)
• On the Convergence of Locally Adaptive and Scalable Diffusion-Based Sampling Methods for Deep Bayesian Neural Network Posteriors [2.3265565167163906]
  Bayesian neural networks are a promising approach for modeling uncertainties in deep neural networks. generating samples from the posterior distribution of neural networks is a major challenge. One advance in that direction would be the incorporation of adaptive step sizes into Monte Carlo Markov chain sampling algorithms. In this paper, we demonstrate that these methods can have a substantial bias in the distribution they sample, even in the limit of vanishing step sizes and at full batch size.
  arXiv  Detail & Related papers  (2024-03-13T15:21:14Z)
• Spike-and-slab shrinkage priors for structurally sparse Bayesian neural networks [0.16385815610837165]
  Sparse deep learning addresses challenges by recovering a sparse representation of the underlying target function. Deep neural architectures compressed via structured sparsity provide low latency inference, higher data throughput, and reduced energy consumption. We propose structurally sparse Bayesian neural networks which prune excessive nodes with (i) Spike-and-Slab Group Lasso (SS-GL), and (ii) Spike-and-Slab Group Horseshoe (SS-GHS) priors.
  arXiv  Detail & Related papers  (2023-08-17T17:14:18Z)
• Bayesian Interpolation with Deep Linear Networks [92.1721532941863]
  Characterizing how neural network depth, width, and dataset size jointly impact model quality is a central problem in deep learning theory. We show that linear networks make provably optimal predictions at infinite depth. We also show that with data-agnostic priors, Bayesian model evidence in wide linear networks is maximized at infinite depth.
  arXiv  Detail & Related papers  (2022-12-29T20:57:46Z)
• Layer Ensembles [95.42181254494287]
  We introduce a method for uncertainty estimation that considers a set of independent categorical distributions for each layer of the network. We show that the method can be further improved by ranking samples, resulting in models that require less memory and time to run.
  arXiv  Detail & Related papers  (2022-10-10T17:52:47Z)
• Wide Bayesian neural networks have a simple weight posterior: theory and accelerated sampling [48.94555574632823]
  Repriorisation transforms a Bayesian neural network (BNN) posterior to a distribution whose KL divergence to the BNN prior vanishes as layer widths grow. We develop a Markov chain Monte Carlo (MCMC) posterior sampling algorithm which mixes faster the wider the BNN. We observe up to 50x higher effective sample size relative to no reparametrisation for both fully-connected and residual networks.
  arXiv  Detail & Related papers  (2022-06-15T17:11:08Z)
• Precise characterization of the prior predictive distribution of deep ReLU networks [45.46732383818331]
  We derive a precise characterization of the prior predictive distribution of finite-width ReLU networks with Gaussian weights. Our results provide valuable guidance on prior design, for instance, controlling the predictive variance with depth- and width-informed priors on the weights of the network.
  arXiv  Detail & Related papers  (2021-06-11T21:21:52Z)
• What Are Bayesian Neural Network Posteriors Really Like? [63.950151520585024]
  We show that Hamiltonian Monte Carlo can achieve significant performance gains over standard and deep ensembles. We also show that deep distributions are similarly close to HMC as standard SGLD, and closer than standard variational inference.
  arXiv  Detail & Related papers  (2021-04-29T15:38:46Z)
• Sampling-free Variational Inference for Neural Networks with Multiplicative Activation Noise [51.080620762639434]
  We propose a more efficient parameterization of the posterior approximation for sampling-free variational inference. Our approach yields competitive results for standard regression problems and scales well to large-scale image classification tasks.
  arXiv  Detail & Related papers  (2021-03-15T16:16:18Z)
• Bayesian Deep Learning and a Probabilistic Perspective of Generalization [56.69671152009899]
  We show that deep ensembles provide an effective mechanism for approximate Bayesian marginalization. We also propose a related approach that further improves the predictive distribution by marginalizing within basins of attraction.
  arXiv  Detail & Related papers  (2020-02-20T15:13:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.