Width-Robust Learnability in Mean-Field Bayesian Neural Networks

• URL: http://arxiv.org/abs/2607.05735v1
• Date: Tue, 07 Jul 2026 01:41:42 GMT
• Title: Width-Robust Learnability in Mean-Field Bayesian Neural Networks
• Abstract summary: We study this question for Bayesian neural networks at the mean-field, or critical feature-learning, scaling.<n>The central quantity is the reduced entropy [ s_infty(y,varepsilon)=limsup_N -frac1Nlog _N0(Lle varepsilon), ] the intensive prior cost of representing a target function.
• Score: 0.08594140167290099
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Infinite-width limits are a standard way to reason about neural networks, but it is not automatic that the limiting learner has the same complexity-theoretic inductive bias as large finite networks. We study this question for Bayesian neural networks at the mean-field, or critical feature-learning, scaling. The central quantity is the \emph{reduced entropy} \[ s_\infty(y,\varepsilon)=\limsup_N -\frac{1}{N}\log π_N^0(L\le \varepsilon), \] the intensive prior cost of representing a target function $y$ to population mean-squared error $\varepsilon$. Our main result is a width-robust learnability theorem. At fixed depth, a family of Boolean-cube targets is learnable from polynomially many samples at infinite width if and only if it is learnable at polynomial width, if and only if its reduced entropy is polynomially bounded. Equivalently, up to polynomial slack in accuracy, the Bayesian mean-field learner generalizes exactly on the targets that can be represented by polynomial-size networks. The forward direction is proved by a form of subsampling: from the infinitely many hidden neurons in the mean-field solution, one can select polynomially many representatives and still preserve the learned function on every input simultaneously. At the critical scaling this subsampling has both an ``active'' component, which keeps the data-dependent low-dimensional statistics, and a ``lazy'' component, which resamples the entropy-dominated directions from the prior. Thus the infinite-width mean-field limit gives a clean analytic description of learning without introducing spurious width-dependent generalization power.

Related papers

• Uncertainty Quantification From Scaling Laws in Deep Neural Networks [0.0]
  Quantifying uncertainty from machine learning analyses is critical to their use in the physical sciences.<n>We compute the mean $mu_mathcalL$ and variance $sigma_mathcalL$ for an ensemble of multi-layer perceptrons.<n>We compare empirically to the results from finite-width networks for three example tasks: MNIST classification, CIFAR classification and calorimeter energy regression.
  arXiv  Detail & Related papers  (2025-03-07T21:15:11Z)
• Learning with Norm Constrained, Over-parameterized, Two-layer Neural Networks [54.177130905659155]
  Recent studies show that a reproducing kernel Hilbert space (RKHS) is not a suitable space to model functions by neural networks. In this paper, we study a suitable function space for over- parameterized two-layer neural networks with bounded norms.
  arXiv  Detail & Related papers  (2024-04-29T15:04:07Z)
• Quantitative CLTs in Deep Neural Networks [12.845031126178593]
  We study the distribution of a fully connected neural network with random Gaussian weights and biases. We obtain quantitative bounds on normal approximations valid at large but finite $n$ and any fixed network depth. Our bounds are strictly stronger in terms of their dependence on network width than any previously available in the literature.
  arXiv  Detail & Related papers  (2023-07-12T11:35:37Z)
• Expressivity of Shallow and Deep Neural Networks for Polynomial Approximation [0.0]
  We establish an exponential lower bound on the complexity of any shallow network approximating the product function over a general compact domain. We also demonstrate this lower bound doesn't apply to normalized Lipschitz monomials over the unit cube.
  arXiv  Detail & Related papers  (2023-03-06T23:01:53Z)
• The Onset of Variance-Limited Behavior for Networks in the Lazy and Rich Regimes [75.59720049837459]
  We study the transition from infinite-width behavior to this variance limited regime as a function of sample size $P$ and network width $N$. We find that finite-size effects can become relevant for very small datasets on the order of $P* sim sqrtN$ for regression with ReLU networks.
  arXiv  Detail & Related papers  (2022-12-23T04:48:04Z)
• Spectral Bias Outside the Training Set for Deep Networks in the Kernel Regime [0.0]
  We show that the network is biased to learn the top eigenfunctions of the Neural Kernel not just on the training set but over the entire input space. This bias depends on the model architecture and input distribution alone. We conclude that local capacity control from the low effective rank of the Fisher Information Matrix is still underexplored theoretically.
  arXiv  Detail & Related papers  (2022-06-06T22:09:15Z)
• The Separation Capacity of Random Neural Networks [78.25060223808936]
  We show that a sufficiently large two-layer ReLU-network with standard Gaussian weights and uniformly distributed biases can solve this problem with high probability. We quantify the relevant structure of the data in terms of a novel notion of mutual complexity.
  arXiv  Detail & Related papers  (2021-07-31T10:25:26Z)
• Redundant representations help generalization in wide neural networks [71.38860635025907]
  We study the last hidden layer representations of various state-of-the-art convolutional neural networks. We find that if the last hidden representation is wide enough, its neurons tend to split into groups that carry identical information, and differ from each other only by statistically independent noise.
  arXiv  Detail & Related papers  (2021-06-07T10:18:54Z)
• Robust Implicit Networks via Non-Euclidean Contractions [63.91638306025768]
  Implicit neural networks show improved accuracy and significant reduction in memory consumption. They can suffer from ill-posedness and convergence instability. This paper provides a new framework to design well-posed and robust implicit neural networks.
  arXiv  Detail & Related papers  (2021-06-06T18:05:02Z)
• Finite Versus Infinite Neural Networks: an Empirical Study [69.07049353209463]
  kernel methods outperform fully-connected finite-width networks. Centered and ensembled finite networks have reduced posterior variance. Weight decay and the use of a large learning rate break the correspondence between finite and infinite networks.
  arXiv  Detail & Related papers  (2020-07-31T01:57:47Z)
• Logarithmic Pruning is All You Need [30.330326149079326]
  Lottery Ticket Hypothesis: Every large neural network contains a subnetwork that, when trained in isolation, achieves comparable performance to the large network. This latter result, however, relies on a number of strong assumptions and guarantees a factor on the size of the large network compared to the target function.
  arXiv  Detail & Related papers  (2020-06-22T11:42:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.