Optimal scaling laws in learning hierarchical multi-index models

• URL: http://arxiv.org/abs/2602.05846v1
• Date: Thu, 05 Feb 2026 16:33:51 GMT
• Title: Optimal scaling laws in learning hierarchical multi-index models
• Abstract summary: We derive exact information-theoretic scaling laws for subspace recovery and prediction error.<n>We show that these optimal rates are achieved by a simple, target-agnostic spectral estimator.
• Score: 28.495611422425238
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this work, we provide a sharp theory of scaling laws for two-layer neural networks trained on a class of hierarchical multi-index targets, in a genuinely representation-limited regime. We derive exact information-theoretic scaling laws for subspace recovery and prediction error, revealing how the hierarchical features of the target are sequentially learned through a cascade of phase transitions. We further show that these optimal rates are achieved by a simple, target-agnostic spectral estimator, which can be interpreted as the small learning-rate limit of gradient descent on the first-layer weights. Once an adapted representation is identified, the readout can be learned statistically optimally, using an efficient procedure. As a consequence, we provide a unified and rigorous explanation of scaling laws, plateau phenomena, and spectral structure in shallow neural networks trained on such hierarchical targets.

Related papers

• SGD in Multiclass Logistic Regression: Sequential Learning and Scaling Laws [55.81298869922287]
  We study the training dynamics of multiclass logistic regression on high-dimensional Gaussian mixture models with a large number of classes.<n>We establish precise scaling laws governing the cross-entropy risk under gradient-based optimization.
  arXiv  Detail & Related papers  (2026-09-07T18:30:09Z)
• Balancing Learning Rates Across Layers: Exact Two-Step Dynamics and Optimal Scaling in Linear Neural Networks [40.02433292363059]
  We study optimal learning-rate selection in two-layer and three-layer linear neural networks trained to learn linear target functions.<n>We derive the exact closed-form expressions for the gradients and test loss after one and two steps of gradient descent.
  arXiv  Detail & Related papers  (2026-05-29T20:28:52Z)
• Rethinking Neural Network Learning Rates: A Stackelberg Perspective [14.206253864662324]
  We show that training neural networks with a smaller learning rate for the body layers and a larger learning rate for the final layer can be interpreted as a two-time-scale alternating descent gradient algorithm.<n>We identify two mechanisms by which non-uniform learning rates can outperform uniform learning rates.
  arXiv  Detail & Related papers  (2026-05-15T01:59:10Z)
• Scaling Laws from Sequential Feature Recovery: A Solvable Hierarchical Model [23.79711176655533]
  We study a globally high-degree function represented by a combination of latent compositional features whose weights decrease as a power law.<n>We show that a layer-wise spectral algorithm adapted to this structure achieves improved scaling relative to shallow, non-adaptive methods.<n>We prove sharp feature-wise recovery thresholds and show that aggregating these transitions yields an explicit power-law decay of the prediction error.
  arXiv  Detail & Related papers  (2026-05-14T08:37:28Z)
• On Neural Scaling Laws for Weather Emulation through Continual Training [76.70310203875866]
  We study neural scaling in Scientific Machine Learning, focusing on models for weather forecasting.<n>To analyze scaling behavior in as simple a setting as possible, we adopt a minimal, scalable, general-purpose Swin Transformer architecture.<n>We show that models trained in this minimalist way follow predictable scaling trends and even outperform standard cosine learning rate schedules.
  arXiv  Detail & Related papers  (2026-03-26T17:37:25Z)
• Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regime [34.77547342230355]
  We present a systematic analysis of scaling laws for quadratic and diagonal neural networks in the feature learning regime.<n>We derive a detailed phase diagram for the scaling exponents of the excess risk as a function of sample complexity and weight decay.<n>We provide a theoretical validation of recent empirical observations connecting the emergence of power-law tails in the weight spectrum with network generalization performance.
  arXiv  Detail & Related papers  (2025-09-29T14:58:13Z)
• Towards the Training of Deeper Predictive Coding Neural Networks [44.14001498773255]
  Predictive coding networks are neural models that perform inference through an iterative energy minimization process.<n>While effective in shallow architectures, they suffer significant performance degradation beyond five to seven layers.<n>We show that this degradation is caused by exponentially imbalanced errors between layers during weight updates, and by predictions from the previous layers not being effective in guiding updates in deeper layers.
  arXiv  Detail & Related papers  (2025-06-30T12:44:47Z)
• Approximating Latent Manifolds in Neural Networks via Vanishing Ideals [20.464009622419766]
  We establish a connection between manifold learning and computational algebra by demonstrating how vanishing ideals can characterize the latent manifold of deep networks.<n>We propose a new neural architecture that truncates a pretrained network at an intermediate layer, and approximates each class manifold via generators of the vanishing ideal.<n>The resulting models have significantly fewer layers than their pretrained baselines, while maintaining comparable accuracy, achieving higher throughput and utilizing fewer parameters.
  arXiv  Detail & Related papers  (2025-02-20T21:23:02Z)
• On Excess Risk Convergence Rates of Neural Network Classifiers [8.329456268842227]
  We study the performance of plug-in classifiers based on neural networks in a binary classification setting as measured by their excess risks. We analyze the estimation and approximation properties of neural networks to obtain a dimension-free, uniform rate of convergence.
  arXiv  Detail & Related papers  (2023-09-26T17:14:10Z)
• Globally Optimal Training of Neural Networks with Threshold Activation Functions [63.03759813952481]
  We study weight decay regularized training problems of deep neural networks with threshold activations. We derive a simplified convex optimization formulation when the dataset can be shattered at a certain layer of the network.
  arXiv  Detail & Related papers  (2023-03-06T18:59:13Z)
• An Adaptive and Stability-Promoting Layerwise Training Approach for Sparse Deep Neural Network Architecture [0.0]
  This work presents a two-stage adaptive framework for developing deep neural network (DNN) architectures that generalize well for a given training data set. In the first stage, a layerwise training approach is adopted where a new layer is added each time and trained independently by freezing parameters in the previous layers. We introduce a epsilon-delta stability-promoting concept as a desirable property for a learning algorithm and show that employing manifold regularization yields a epsilon-delta stability-promoting algorithm.
  arXiv  Detail & Related papers  (2022-11-13T09:51:16Z)
• Scaling Forward Gradient With Local Losses [117.22685584919756]
  Forward learning is a biologically plausible alternative to backprop for learning deep neural networks. We show that it is possible to substantially reduce the variance of the forward gradient by applying perturbations to activations rather than weights. Our approach matches backprop on MNIST and CIFAR-10 and significantly outperforms previously proposed backprop-free algorithms on ImageNet.
  arXiv  Detail & Related papers  (2022-10-07T03:52:27Z)
• Optimization-Based Separations for Neural Networks [57.875347246373956]
  We show that gradient descent can efficiently learn ball indicator functions using a depth 2 neural network with two layers of sigmoidal activations. This is the first optimization-based separation result where the approximation benefits of the stronger architecture provably manifest in practice.
  arXiv  Detail & Related papers  (2021-12-04T18:07:47Z)
• Subquadratic Overparameterization for Shallow Neural Networks [60.721751363271146]
  We provide an analytical framework that allows us to adopt standard neural training strategies. We achieve the desiderata viaak-Lojasiewicz, smoothness, and standard assumptions.
  arXiv  Detail & Related papers  (2021-11-02T20:24:01Z)
• Proxy Convexity: A Unified Framework for the Analysis of Neural Networks Trained by Gradient Descent [95.94432031144716]
  We propose a unified non- optimization framework for the analysis of a learning network. We show that existing guarantees can be trained unified through gradient descent.
  arXiv  Detail & Related papers  (2021-06-25T17:45:00Z)
• Deep Networks from the Principle of Rate Reduction [32.87280757001462]
  This work attempts to interpret modern deep (convolutional) networks from the principles of rate reduction and (shift) invariant classification. We show that the basic iterative ascent gradient scheme for optimizing the rate reduction of learned features naturally leads to a multi-layer deep network, one iteration per layer. All components of this "white box" network have precise optimization, statistical, and geometric interpretation.
  arXiv  Detail & Related papers  (2020-10-27T06:01:43Z)
• Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Loss [0.0]
  Neural networks trained to minimize the logistic (a.k.a. cross-entropy) loss with gradient-based methods are observed to perform well in many supervised classification tasks. We analyze the training and generalization behavior of infinitely wide two-layer neural networks with homogeneous activations.
  arXiv  Detail & Related papers  (2020-02-11T15:42:09Z)
• Large Batch Training Does Not Need Warmup [111.07680619360528]
  Training deep neural networks using a large batch size has shown promising results and benefits many real-world applications. In this paper, we propose a novel Complete Layer-wise Adaptive Rate Scaling (CLARS) algorithm for large-batch training. Based on our analysis, we bridge the gap and illustrate the theoretical insights for three popular large-batch training techniques.
  arXiv  Detail & Related papers  (2020-02-04T23:03:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.