Singular perturbations and hierarchical learning in two-layer neural networks

• URL: http://arxiv.org/abs/2607.10869v1
• Date: Sun, 12 Jul 2026 18:10:23 GMT
• Title: Singular perturbations and hierarchical learning in two-layer neural networks
• Abstract summary: We study the population gradient flow of an infinitely wide two-layer neural network learning a misspecified single-index model in high dimension.<n>We prove that the constant and linear components of the hidden link function are indeed recovered within the predicted timescales.
• Score: 3.899493143383179
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We study the population gradient flow of an infinitely wide two-layer neural network learning a misspecified single-index model in high dimension. The two layers are optimized jointly, with a perturbative parameter tuning the relative training speed between the first and second layer. This setting was considered by Berthier, Montanari and Zhou in \cite{berthier2024learning}, who conjectured a hierarchical learning scenario with explicit timescales as the second layer is trained faster than the first. In this paper, we prove that the constant and linear components of the hidden link function are indeed recovered within the predicted timescales, at sharp explicit thresholds. We then analyze the onset of learning of the quadratic component and show that the components learned at earlier stages continue to influence the dynamics in an essential way. Our proof is based on quantitative approximation results for singularly perturbed flows evolving near a manifold defined by integral constraints. At a phenomenological level, we also show that the empirical measure of the weights displays singular behaviour when reaching the quadratic component of the hidden link, with a small fraction of neurons growing significantly while the remaining ones rearrange to preserve the components already learned.

Related papers

• Learning Orthogonal Multi-Index Models Beyond Small Initialization: Incremental Learning, Competitive Dynamics and Symmetry [52.75288960194919]
  Recent work has identified incremental learning in shallow networks trained on single-index and multi-index models.<n>We introduce a symmetry-based finite-width approximation via symmetrized networks, rather than comparing directly with an infinite-width limit.
  arXiv  Detail & Related papers  (2026-09-09T22:35:31Z)
• Dichotomy of Feature Learning and Unlearning: Fast-Slow Analysis on Neural Networks with Stochastic Gradient Descent [5.5165579223151795]
  In this study, we consider the infinite-width limit of a two-layer neural network updated with a large-batch gradient, then derive differential equations with different time scales.<n>The direction of a flow on a critical manifold, determined by the slow dynamics, decides whether feature unlearning occurs.<n>Our results yield the following insights: (i) the strength of the primary nonlinear term in data induces the feature unlearning, and (ii) an initial scale of the second-layer weights mitigates the feature unlearning.
  arXiv  Detail & Related papers  (2026-02-07T05:50:44Z)
• Explaining Grokking and Information Bottleneck through Neural Collapse Emergence [33.22494588674352]
  We present a unified explanation of grokking and the information bottleneck principle through the lens of neural collapse.<n>We show that the contraction of population within-class variance is a key factor underlying both grokking and information bottleneck.<n>By analyzing the dynamics of neural collapse, we show that distinct time scales between fitting the training set and the progression of neural collapse account for the behavior of the late-phase phenomena.
  arXiv  Detail & Related papers  (2025-09-25T07:17:41Z)
• New Evidence of the Two-Phase Learning Dynamics of Neural Networks [59.55028392232715]
  We introduce an interval-wise perspective that compares network states across a time window.<n>We show that the response of the network to a perturbation exhibits a transition from chaotic to stable.<n>We also find that after this transition point the model's functional trajectory is confined to a narrow cone-shaped subset.
  arXiv  Detail & Related papers  (2025-05-20T04:03:52Z)
• A Theory of Non-Linear Feature Learning with One Gradient Step in Two-Layer Neural Networks [43.281323350357404]
  Feature learning is thought to be one of the fundamental reasons for the success of deep neural networks.<n>We show that with a learning rate that grows with the sample size, such training in fact introduces multiple rank-one components.
  arXiv  Detail & Related papers  (2023-10-11T20:55:02Z)
• Learning time-scales in two-layers neural networks [11.878594839685471]
  We study the gradient flow dynamics of a wide two-layer neural network in high-dimension.<n>Based on new rigorous results, we propose a scenario for the learning dynamics in this setting.
  arXiv  Detail & Related papers  (2023-02-28T19:52:26Z)
• Catapult Dynamics and Phase Transitions in Quadratic Nets [15.388101504755085]
  We will prove that the catapult phase exists in a large class of models, including quadratic models and two-layer, homogenous neural nets.<n>We show that for a certain range of learning rates the weight norm decreases whenever the loss becomes large.<n>We also empirically study learning rates beyond this theoretically derived range and show that the activation map of ReLU nets trained with super-critical learning rates becomes increasingly sparse as we increase the learning rate.
  arXiv  Detail & Related papers  (2023-01-18T19:03:48Z)
• Improved Convergence Guarantees for Shallow Neural Networks [91.3755431537592]
  We prove convergence of depth 2 neural networks, trained via gradient descent, to a global minimum. Our model has the following features: regression with quadratic loss function, fully connected feedforward architecture, RelU activations, Gaussian data instances, adversarial labels. They strongly suggest that, at least in our model, the convergence phenomenon extends well beyond the NTK regime''
  arXiv  Detail & Related papers  (2022-12-05T14:47:52Z)
• Multi-scale Feature Learning Dynamics: Insights for Double Descent [71.91871020059857]
  We study the phenomenon of "double descent" of the generalization error. We find that double descent can be attributed to distinct features being learned at different scales.
  arXiv  Detail & Related papers  (2021-12-06T18:17:08Z)
• Optimization-Based Separations for Neural Networks [57.875347246373956]
  We show that gradient descent can efficiently learn ball indicator functions using a depth 2 neural network with two layers of sigmoidal activations. This is the first optimization-based separation result where the approximation benefits of the stronger architecture provably manifest in practice.
  arXiv  Detail & Related papers  (2021-12-04T18:07:47Z)
• Towards an Understanding of Benign Overfitting in Neural Networks [104.2956323934544]
  Modern machine learning models often employ a huge number of parameters and are typically optimized to have zero training loss. We examine how these benign overfitting phenomena occur in a two-layer neural network setting. We show that it is possible for the two-layer ReLU network interpolator to achieve a near minimax-optimal learning rate.
  arXiv  Detail & Related papers  (2021-06-06T19:08:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.