On the Interplay of Priors and Overparametrization in Bayesian Neural Network Posteriors

• URL: http://arxiv.org/abs/2603.22030v1
• Date: Mon, 23 Mar 2026 14:36:13 GMT
• Title: On the Interplay of Priors and Overparametrization in Bayesian Neural Network Posteriors
• Abstract summary: We study how overparametrization and priors together reshape BNN posteriors.<n>We show that redundancy introduces three key phenomena that fundamentally reshape the posterior geometry.
• Score: 12.825948364196057
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Bayesian neural network (BNN) posteriors are often considered impractical for inference, as symmetries fragment them, non-identifiabilities inflate dimensionality, and weight-space priors are seen as meaningless. In this work, we study how overparametrization and priors together reshape BNN posteriors and derive implications allowing us to better understand their interplay. We show that redundancy introduces three key phenomena that fundamentally reshape the posterior geometry: balancedness, weight reallocation on equal-probability manifolds, and prior conformity. We validate our findings through extensive experiments with posterior sampling budgets that far exceed those of earlier works, and demonstrate how overparametrization induces structured, prior-aligned weight posterior distributions.

Related papers

• ANCRe: Adaptive Neural Connection Reassignment for Efficient Depth Scaling [57.91760520589592]
  Scaling network depth has been a central driver behind the success of modern foundation models.<n>This paper revisits the default mechanism for deepening neural networks, namely residual connections.<n>We introduce adaptive neural connection reassignment (ANCRe), a principled and lightweight framework that parameterizes and learns residual connectivities from the data.
  arXiv  Detail & Related papers  (2026-02-09T18:54:18Z)
• VIKING: Deep variational inference with stochastic projections [48.946143517489496]
  Variational mean field approximations tend to struggle with contemporary overparametrized deep neural networks.<n>We propose a simple variational family that considers two independent linear subspaces of the parameter space.<n>This allows us to build a fully-correlated approximate posterior reflecting the overparametrization.
  arXiv  Detail & Related papers  (2025-10-27T15:38:35Z)
• Posterior Contraction for Sparse Neural Networks in Besov Spaces with Intrinsic Dimensionality [12.055886881947577]
  This work establishes that sparse Bayesian neural networks achieve optimal posterior contraction rates over anisotropic Besov spaces and their hierarchical compositions.<n>We show that these priors enable rate adaptation, allowing the posterior to contract at the optimal rate even when the smoothness level of the true function is unknown.
  arXiv  Detail & Related papers  (2025-06-23T21:29:40Z)
• Variational Inference Failures Under Model Symmetries: Permutation Invariant Posteriors for Bayesian Neural Networks [43.88179780450706]
  We investigate the impact of weight space permutation symmetries on variational inference. We devise a symmetric symmetrization mechanism for constructing permutation invariant variational posteriors. We show that the symmetrized distribution has a strictly better fit to the true posterior, and that it can be trained using the original ELBO objective.
  arXiv  Detail & Related papers  (2024-08-10T09:06:34Z)
• A Symmetry-Aware Exploration of Bayesian Neural Network Posteriors [5.54475507578913]
  The distribution of the weights of modern deep neural networks (DNNs) is an eminently complex object due to its extremely high dimensionality. This paper proposes one of the first large-scale explorations of the posterior distribution of BNNs, expanding its study to real-world vision tasks and architectures.
  arXiv  Detail & Related papers  (2023-10-12T12:45:13Z)
• Joint Bayesian Inference of Graphical Structure and Parameters with a Single Generative Flow Network [59.79008107609297]
  We propose in this paper to approximate the joint posterior over the structure of a Bayesian Network. We use a single GFlowNet whose sampling policy follows a two-phase process. Since the parameters are included in the posterior distribution, this leaves more flexibility for the local probability models.
  arXiv  Detail & Related papers  (2023-05-30T19:16:44Z)
• Bayesian Interpolation with Deep Linear Networks [92.1721532941863]
  Characterizing how neural network depth, width, and dataset size jointly impact model quality is a central problem in deep learning theory. We show that linear networks make provably optimal predictions at infinite depth. We also show that with data-agnostic priors, Bayesian model evidence in wide linear networks is maximized at infinite depth.
  arXiv  Detail & Related papers  (2022-12-29T20:57:46Z)
• Wide Bayesian neural networks have a simple weight posterior: theory and accelerated sampling [48.94555574632823]
  Repriorisation transforms a Bayesian neural network (BNN) posterior to a distribution whose KL divergence to the BNN prior vanishes as layer widths grow. We develop a Markov chain Monte Carlo (MCMC) posterior sampling algorithm which mixes faster the wider the BNN. We observe up to 50x higher effective sample size relative to no reparametrisation for both fully-connected and residual networks.
  arXiv  Detail & Related papers  (2022-06-15T17:11:08Z)
• Bayesian neural network priors for edge-preserving inversion [3.2046720177804646]
  A class of prior distributions based on the output of neural networks with heavy-tailed weights is introduced. We show theoretically that samples from such priors have desirable discontinuous-like properties even when the network width is finite.
  arXiv  Detail & Related papers  (2021-12-20T16:39:05Z)
• Revisiting Initialization of Neural Networks [72.24615341588846]
  We propose a rigorous estimation of the global curvature of weights across layers by approximating and controlling the norm of their Hessian matrix. Our experiments on Word2Vec and the MNIST/CIFAR image classification tasks confirm that tracking the Hessian norm is a useful diagnostic tool.
  arXiv  Detail & Related papers  (2020-04-20T18:12:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.