How Width and Data Shape Generalization Scaling Laws in Quadratic Neural Networks

• URL: http://arxiv.org/abs/2606.28242v1
• Date: Fri, 26 Jun 2026 16:27:46 GMT
• Title: How Width and Data Shape Generalization Scaling Laws in Quadratic Neural Networks
• Abstract summary: We study how generalization scales with the number of trainable parameters and the number of samples in a feature-learning model.<n>Our results reveal a phase diagram with distinct scaling regimes as the number of parameters varies.
• Score: 29.557636603874105
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Understanding how performance scales jointly with model size and data is a central problem in modern machine learning. Existing theoretical works on scaling laws typically describe generalization as a function of data or compute, often in fixed-feature or infinite-width regimes and for online SGD. Here, we instead study how generalization scales with the number of trainable parameters and the number of samples in a feature-learning model. We analyze $\ell_2$-regularized empirical test error minimization in a quadratic two-layer network in a finite-sample setting with structured data. This setting allows for an explicit characterization of the generalization error as a function of the number of samples, model width, and regularization. Our results reveal a phase diagram with distinct scaling regimes as the number of parameters varies. In particular, the generalization error follows data-dependent power laws controlled by the spectral structure of the target. We further characterize the transitions between regimes, including the onset of interpolation, and their impact on generalization.

Related papers

• A renormalization-group inspired lattice-based framework for piecewise generalized linear models [0.6599344783327052]
  We introduce a class of models inspired by renormalization group (RG) theory.<n>They are built on additive hierarchical expansions analogous to those appearing in functional ANOVA and mixed-effects models.<n>We use techniques from statistical physics -- specifically replica analysis -- to study their generalization properties.
  arXiv  Detail & Related papers  (2026-05-06T22:27:50Z)
• Compute-Optimal LLMs Provably Generalize Better With Scale [102.29926217670926]
  We develop generalization bounds on the pretraining objective of large language models (LLMs) in the compute-optimal regime.<n>We introduce a novel, fully empirical Freedman-type martingale concentration that tightens existing bounds by accounting for the variance of the loss function.<n>We produce a scaling law for the generalization gap, with bounds that become predictably stronger with scale.
  arXiv  Detail & Related papers  (2025-04-21T16:26:56Z)
• Analyzing Neural Scaling Laws in Two-Layer Networks with Power-Law Data Spectra [0.0]
  Neural scaling laws describe how the performance of deep neural networks scales with key factors such as training data size, model complexity, and training time. We employ techniques from statistical mechanics to analyze one-pass gradient descent within a student-teacher framework.
  arXiv  Detail & Related papers  (2024-10-11T17:21:42Z)
• Scaling and renormalization in high-dimensional regression [72.59731158970894]
  We present a unifying perspective on recent results on ridge regression.<n>We use the basic tools of random matrix theory and free probability, aimed at readers with backgrounds in physics and deep learning.<n>Our results extend and provide a unifying perspective on earlier models of scaling laws.
  arXiv  Detail & Related papers  (2024-05-01T15:59:00Z)
• What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalization [111.55277952086155]
  We study In-Context Learning (ICL) by addressing several open questions. We show that, without updating the neural network parameters, ICL implicitly implements the Bayesian model averaging algorithm. We prove that the error of pretrained model is bounded by a sum of an approximation error and a generalization error.
  arXiv  Detail & Related papers  (2023-05-30T21:23:47Z)
• Instance-Dependent Generalization Bounds via Optimal Transport [51.71650746285469]
  Existing generalization bounds fail to explain crucial factors that drive the generalization of modern neural networks. We derive instance-dependent generalization bounds that depend on the local Lipschitz regularity of the learned prediction function in the data space. We empirically analyze our generalization bounds for neural networks, showing that the bound values are meaningful and capture the effect of popular regularization methods during training.
  arXiv  Detail & Related papers  (2022-11-02T16:39:42Z)
• A Solvable Model of Neural Scaling Laws [72.8349503901712]
  Large language models with a huge number of parameters, when trained on near internet-sized number of tokens, have been empirically shown to obey neural scaling laws. We propose a statistical model -- a joint generative data model and random feature model -- that captures this neural scaling phenomenology. Key findings are the manner in which the power laws that occur in the statistics of natural datasets are extended by nonlinear random feature maps.
  arXiv  Detail & Related papers  (2022-10-30T15:13:18Z)
• Studying Generalization Through Data Averaging [0.0]
  We study train and test performance, as well as the generalization gap given by the mean of their difference over different data set samples. We predict some aspects about how the generalization gap and model train and test performance vary as a function of SGD noise.
  arXiv  Detail & Related papers  (2022-06-28T00:03:40Z)
• Post-mortem on a deep learning contest: a Simpson's paradox and the complementary roles of scale metrics versus shape metrics [61.49826776409194]
  We analyze a corpus of models made publicly-available for a contest to predict the generalization accuracy of neural network (NN) models. We identify what amounts to a Simpson's paradox: where "scale" metrics perform well overall but perform poorly on sub partitions of the data. We present two novel shape metrics, one data-independent, and the other data-dependent, which can predict trends in the test accuracy of a series of NNs.
  arXiv  Detail & Related papers  (2021-06-01T19:19:49Z)
• Explaining Neural Scaling Laws [17.115592382420626]
  Population loss of trained deep neural networks often follows precise power-law scaling relations. We propose a theory that explains the origins of and connects these scaling laws. We identify variance-limited and resolution-limited scaling behavior for both dataset and model size.
  arXiv  Detail & Related papers  (2021-02-12T18:57:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.