Sharp feature-learning transitions and Bayes-optimal neural scaling laws in extensive-width networks

• URL: http://arxiv.org/abs/2605.10395v1
• Date: Mon, 11 May 2026 11:39:03 GMT
• Title: Sharp feature-learning transitions and Bayes-optimal neural scaling laws in extensive-width networks
• Abstract summary: We study the information-theoretic limits of learning a one-hidden-layer teacher network with hierarchical features from noisy queries.<n>We show that a student trained with textscAdam near the effective width $k_c$ achieves these optimal scaling laws.
• Score: 8.250374560598493
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We study the information-theoretic limits of learning a one-hidden-layer teacher network with hierarchical features from noisy queries, in the context of knowledge transfer to a smaller student model. We work in the high-dimensional regime where the teacher width $k$ scales linearly with the input dimension $d$ -- a setting that captures large-but-finite-width networks and has only recently become analytically tractable. Using a heuristic leave-one-out decoupling argument, validated numerically throughout, we derive asymptotically sharp characterizations of the Bayes-optimal generalization error and individual feature overlaps via a system of closed fixed-point equations. These equations reveal that feature learnability is governed by a sequence of sharp phase transitions: as data grows, teacher features become recoverable sequentially, each through a discontinuous jump in overlap. This sequential acquisition underlies a precise notion of \textit{effective width} $k_c$ -- the number of learnable features at a given data budget $n$ -- which unifies two distinct scaling regimes: a feature-learning regime in which the Bayes-optimal generalization error $\varepsilon^{\rm BO}$ scales as $ n^{1/(2β)-1}$, and a refinement regime in which it scales as $n^{-1}$, where $β>1/2$ is the exponent of the power-law feature hierarchy. Both laws collapse to the single relation $\varepsilon^{\rm BO}=Θ(k_c d/n)$. We further show empirically that a student trained with \textsc{Adam} near the effective width $k_c$ achieves these optimal scaling laws (up to a small algorithmic gap), and provide an information-theoretic account of the associated scaling in model size.

Related papers

• A Boundary-Layer Mechanism for One-Third Scaling in Online Softmax Classification [3.058685580689604]
  Hardlabel classification is usually trained with smooth surrogate losses, most prominently softmax crossentropy.<n>We isolate an mechanism by which this mismatch between smooth surrogate and discrete labels produces powerlaw learning curves.<n>We show that learning-rate schedules can improve the generalization error towards a $_g sim -1/2$ power law.
  arXiv  Detail & Related papers  (2026-05-21T11:26:32Z)
• Feature Learning in Linear-Width Two-Layer Networks: Two vs. One Step of Gradient Descent [32.9638210129515]
  We study feature learning in two-layer neural networks within the linear-width regime.<n>By characterizing these early-phase evolutions, our work proposes a tractable framework for studying optimization and feature learning phenomenology.
  arXiv  Detail & Related papers  (2026-05-18T02:37:50Z)
• Neural Networks Learn Generic Multi-Index Models Near Information-Theoretic Limit [66.20349460098275]
  We study the gradient descent learning of a general Gaussian Multi-index model $f(boldsymbolx)=g(boldsymbolUboldsymbolx)$ with hidden subspace $boldsymbolUin mathbbRrtimes d$.<n>We prove that under generic non-degenerate assumptions on the link function, a standard two-layer neural network trained via layer-wise gradient descent can agnostically learn the target with $o_d(1)$ test error.
  arXiv  Detail & Related papers  (2025-11-19T04:46:47Z)
• Beyond Softmax: A Natural Parameterization for Categorical Random Variables [61.709831225296305]
  We introduce the $textitcatnat$ function, a function composed of a sequence of hierarchical binary splits.<n>A rich set of experiments show that the proposed function improves the learning efficiency and yields models characterized by consistently higher test performance.
  arXiv  Detail & Related papers  (2025-09-29T12:55:50Z)
• Emergence and scaling laws in SGD learning of shallow neural networks [64.48316762675141]
  We study the complexity of online gradient descent (SGD) for learning a two-layer neural network with $P$ neurons on isotropic Gaussian data.<n>We provide a precise analysis of SGD dynamics for the training of a student two-layer network to minimize the mean squared error (MSE) objective.
  arXiv  Detail & Related papers  (2025-04-28T16:58:55Z)
• Optimal generalisation and learning transition in extensive-width shallow neural networks near interpolation [4.976898227858662]
  We consider a teacher-student model of supervised learning with a fully-trained two-layer neural network.<n>We provide an effective theory for approximating the Bayes-optimal generalisation error of the network for any activation function.
  arXiv  Detail & Related papers  (2025-01-30T17:56:52Z)
• Learning Orthogonal Multi-Index Models: A Fine-Grained Information Exponent Analysis [54.57279006229212]
  Information exponent has played an important role in predicting the sample complexity of online gradient descent.<n>In this work, we show that by considering both second- and higher-order terms, we can first learn the relevant space using the second-order terms.<n>The overall sample and complexity of online SGD is $tildeO( d PL-1 )$.
  arXiv  Detail & Related papers  (2024-10-13T00:14:08Z)
• Bayesian Inference with Deep Weakly Nonlinear Networks [57.95116787699412]
  We show at a physics level of rigor that Bayesian inference with a fully connected neural network is solvable. We provide techniques to compute the model evidence and posterior to arbitrary order in $1/N$ and at arbitrary temperature.
  arXiv  Detail & Related papers  (2024-05-26T17:08:04Z)
• The Onset of Variance-Limited Behavior for Networks in the Lazy and Rich Regimes [75.59720049837459]
  We study the transition from infinite-width behavior to this variance limited regime as a function of sample size $P$ and network width $N$. We find that finite-size effects can become relevant for very small datasets on the order of $P* sim sqrtN$ for regression with ReLU networks.
  arXiv  Detail & Related papers  (2022-12-23T04:48:04Z)
• High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representation [89.21686761957383]
  We study the first gradient descent step on the first-layer parameters $boldsymbolW$ in a two-layer network. Our results demonstrate that even one step can lead to a considerable advantage over random features.
  arXiv  Detail & Related papers  (2022-05-03T12:09:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.