A Symmetry-Aware Exploration of Bayesian Neural Network Posteriors
- URL: http://arxiv.org/abs/2310.08287v1
- Date: Thu, 12 Oct 2023 12:45:13 GMT
- Title: A Symmetry-Aware Exploration of Bayesian Neural Network Posteriors
- Authors: Olivier Laurent, Emanuel Aldea and Gianni Franchi
- Abstract summary: The distribution of the weights of modern deep neural networks (DNNs) is an eminently complex object due to its extremely high dimensionality.
This paper proposes one of the first large-scale explorations of the posterior distribution of BNNs, expanding its study to real-world vision tasks and architectures.
- Score: 5.54475507578913
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: The distribution of the weights of modern deep neural networks (DNNs) -
crucial for uncertainty quantification and robustness - is an eminently complex
object due to its extremely high dimensionality. This paper proposes one of the
first large-scale explorations of the posterior distribution of deep Bayesian
Neural Networks (BNNs), expanding its study to real-world vision tasks and
architectures. Specifically, we investigate the optimal approach for
approximating the posterior, analyze the connection between posterior quality
and uncertainty quantification, delve into the impact of modes on the
posterior, and explore methods for visualizing the posterior. Moreover, we
uncover weight-space symmetries as a critical aspect for understanding the
posterior. To this extent, we develop an in-depth assessment of the impact of
both permutation and scaling symmetries that tend to obfuscate the Bayesian
posterior. While the first type of transformation is known for duplicating
modes, we explore the relationship between the latter and L2 regularization,
challenging previous misconceptions. Finally, to help the community improve our
understanding of the Bayesian posterior, we will shortly release the first
large-scale checkpoint dataset, including thousands of real-world models and
our codes.
Related papers
- Posterior and variational inference for deep neural networks with heavy-tailed weights [0.0]
We consider deep neural networks in a Bayesian framework with a prior distribution sampling the network weights at random.
We show that the corresponding posterior distribution achieves near-optimal minimax contraction rates.
We also provide variational Bayes counterparts of the results, that show that mean-field variational approximations still benefit from near-optimal theoretical support.
arXiv Detail & Related papers (2024-06-05T15:24:20Z) - Uncertainty-guided Optimal Transport in Depth Supervised Sparse-View 3D Gaussian [49.21866794516328]
3D Gaussian splatting has demonstrated impressive performance in real-time novel view synthesis.
Previous approaches have incorporated depth supervision into the training of 3D Gaussians to mitigate overfitting.
We introduce a novel method to supervise the depth distribution of 3D Gaussians, utilizing depth priors with integrated uncertainty estimates.
arXiv Detail & Related papers (2024-05-30T03:18:30Z) - Bayesian imaging inverse problem with SA-Roundtrip prior via HMC-pCN
sampler [3.717366858126521]
The selection of the prior distribution is learned from, and therefore an important representation learning of, available prior measurements.
The SA-Roundtrip, a novel deep generative prior, is introduced to enable controlled sampling generation and identify the data's intrinsic dimension.
arXiv Detail & Related papers (2023-10-24T17:16:45Z) - Bayesian Interpolation with Deep Linear Networks [92.1721532941863]
Characterizing how neural network depth, width, and dataset size jointly impact model quality is a central problem in deep learning theory.
We show that linear networks make provably optimal predictions at infinite depth.
We also show that with data-agnostic priors, Bayesian model evidence in wide linear networks is maximized at infinite depth.
arXiv Detail & Related papers (2022-12-29T20:57:46Z) - Variational Laplace Autoencoders [53.08170674326728]
Variational autoencoders employ an amortized inference model to approximate the posterior of latent variables.
We present a novel approach that addresses the limited posterior expressiveness of fully-factorized Gaussian assumption.
We also present a general framework named Variational Laplace Autoencoders (VLAEs) for training deep generative models.
arXiv Detail & Related papers (2022-11-30T18:59:27Z) - Wide Bayesian neural networks have a simple weight posterior: theory and
accelerated sampling [48.94555574632823]
Repriorisation transforms a Bayesian neural network (BNN) posterior to a distribution whose KL divergence to the BNN prior vanishes as layer widths grow.
We develop a Markov chain Monte Carlo (MCMC) posterior sampling algorithm which mixes faster the wider the BNN.
We observe up to 50x higher effective sample size relative to no reparametrisation for both fully-connected and residual networks.
arXiv Detail & Related papers (2022-06-15T17:11:08Z) - Probabilistic Mass Mapping with Neural Score Estimation [4.079848600120986]
We introduce a novel methodology for efficient sampling of the high-dimensional Bayesian posterior of the weak lensing mass-mapping problem.
We aim to demonstrate the accuracy of the method on simulations, and then proceed to applying it to the mass reconstruction of the HST/ACS COSMOS field.
arXiv Detail & Related papers (2022-01-14T17:07:48Z) - Deep Two-View Structure-from-Motion Revisited [83.93809929963969]
Two-view structure-from-motion (SfM) is the cornerstone of 3D reconstruction and visual SLAM.
We propose to revisit the problem of deep two-view SfM by leveraging the well-posedness of the classic pipeline.
Our method consists of 1) an optical flow estimation network that predicts dense correspondences between two frames; 2) a normalized pose estimation module that computes relative camera poses from the 2D optical flow correspondences, and 3) a scale-invariant depth estimation network that leverages epipolar geometry to reduce the search space, refine the dense correspondences, and estimate relative depth maps.
arXiv Detail & Related papers (2021-04-01T15:31:20Z) - Revisiting Initialization of Neural Networks [72.24615341588846]
We propose a rigorous estimation of the global curvature of weights across layers by approximating and controlling the norm of their Hessian matrix.
Our experiments on Word2Vec and the MNIST/CIFAR image classification tasks confirm that tracking the Hessian norm is a useful diagnostic tool.
arXiv Detail & Related papers (2020-04-20T18:12:56Z) - Liberty or Depth: Deep Bayesian Neural Nets Do Not Need Complex Weight
Posterior Approximations [40.384018112884874]
We prove that deep mean-field variational weight posteriors can induce similar distributions in function-space to those induced by shallower networks.
Our results suggest that using mean-field variational inference in a deeper model is both a practical and theoretically justified alternative to structured approximations.
arXiv Detail & Related papers (2020-02-10T13:11:45Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.