Neural Quadratic Forms: A Unified Minimal Model for Sudden Learning and Scaling Laws

• URL: http://arxiv.org/abs/2608.13335v1
• Date: Thu, 13 Aug 2026 15:05:43 GMT
• Title: Neural Quadratic Forms: A Unified Minimal Model for Sudden Learning and Scaling Laws
• Abstract summary: Neural networks trained by gradient descent on a smooth cost function can learn in steps.<n>Training losses instead follow smooth power laws.<n>We show that a symmetry fixes what those variables are.
• Score: 12.27037893891717
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural networks trained by gradient descent on a smooth cost function can nevertheless learn in steps: the cost holds on long plateaus and then drops abruptly. Meanwhile, training losses instead follow smooth power laws. Variants of both behaviors occur in architectures with very different microscopic structures, which is the signature of a few relevant collective variables. We show that a symmetry fixes what those variables are: a network layer is a sum over interchangeable units, so relabeling the units leaves it unchanged; given smoothness and the condition that a unit's gradient vanish at the origin, symmetry then enforces a universal leading form for the expansion about the near-zero weights present at the start of training, the quadratic $\Tr[WW^{\top}A(x)]$, in which every architectural detail is confined to a single ``structure matrix" $A(x)$ that we compute for each architecture. Perceptrons, attention layers, mixtures of experts, and convolutions become one model at different $A$. Its training dynamics then close on the ``order parameter" $M=WW^{\top}$ and, whenever the data matrices share an eigenbasis, reduce to a Lotka--Volterra equation whose modes switch on one after another. The smaller the initial weights, the further apart the switch-on times, and the plateaus appear as a singular limit of a smooth flow; when many modes are unresolved the same events merge into a power law in training time whose exponent the theory predicts. We confirm both numerically across training methods and architectures.

Related papers

• Learning Orthogonal Multi-Index Models Beyond Small Initialization: Incremental Learning, Competitive Dynamics and Symmetry [52.75288960194919]
  Recent work has identified incremental learning in shallow networks trained on single-index and multi-index models.<n>We introduce a symmetry-based finite-width approximation via symmetrized networks, rather than comparing directly with an infinite-width limit.
  arXiv  Detail & Related papers  (2026-09-09T22:35:31Z)
• Equivariance Breaks the Learning Rate [24.409404298953234]
  Equivariant networks are commonly trained with Adam.<n>We identify one source of this difference inside equivariant linear layers.<n>We show that blockwise step control and momentum accumulation account for much of Muon's advantage.
  arXiv  Detail & Related papers  (2026-09-08T07:52:03Z)
• Implicit Bias of SGD in Multivariate ReLU Networks: Effective Width Collapse [16.78534503577613]
  We show that the input weights and biases align along finitely many directions, leading to an effective width collapse.<n>We establish a non-redundancy property of the learned representation by proving that each learned direction induces a unique ternary activation pattern on the training data.
  arXiv  Detail & Related papers  (2026-07-03T22:03:13Z)
• Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectors [110.10775872372047]
  We propose emphMagnitude-Direction-Direction (MD) Decoupling.<n>It factorizes each weight into a fixed-norm direction on a hypersphere and learnable per-row and per-column magnitude gains, updated at separate learning rates.<n>Across both Adam and Muon, MD Decoupling improves on well-tuned baselines, transfers the optimal LR across model width without retuning, and continues to help at scale on large Mixture-of-Experts models.
  arXiv  Detail & Related papers  (2026-06-24T15:40:26Z)
• Sharp feature-learning transitions and Bayes-optimal neural scaling laws in extensive-width networks [8.250374560598493]
  We study the information-theoretic limits of learning a one-hidden-layer teacher network with hierarchical features from noisy queries.<n>We show that a student trained with textscAdam near the effective width $k_c$ achieves these optimal scaling laws.
  arXiv  Detail & Related papers  (2026-05-11T11:39:03Z)
• Beyond Softmax: A Natural Parameterization for Categorical Random Variables [61.709831225296305]
  We introduce the $textitcatnat$ function, a function composed of a sequence of hierarchical binary splits.<n>A rich set of experiments show that the proposed function improves the learning efficiency and yields models characterized by consistently higher test performance.
  arXiv  Detail & Related papers  (2025-09-29T12:55:50Z)
• Provable Scaling Laws of Feature Emergence from Learning Dynamics of Grokking [44.614763110719274]
  We study the phenomenon of grokking, i.e., delayed generalization.<n>We propose a novel framework that captures three key stages for the grokking behavior of 2-layer nonlinear networks.<n>Our study sheds lights on roles played by hypers such as weight decay, learning rate and sample sizes in grokking.
  arXiv  Detail & Related papers  (2025-09-25T20:08:09Z)
• Gradient dynamics for low-rank fine-tuning beyond kernels [9.275532709125242]
  We study low-rank fine-tuning in a student-teacher setting. We prove under mild assumptions that a student model which is matrix at the base model and trained with online gradient descent will converge to the teacher.
  arXiv  Detail & Related papers  (2024-11-23T00:00:28Z)
• Neural Metamorphosis [72.88137795439407]
  This paper introduces a new learning paradigm termed Neural Metamorphosis (NeuMeta), which aims to build self-morphable neural networks. NeuMeta directly learns the continuous weight manifold of neural networks. It sustains full-size performance even at a 75% compression rate.
  arXiv  Detail & Related papers  (2024-10-10T14:49:58Z)
• A Dynamical Model of Neural Scaling Laws [79.59705237659547]
  We analyze a random feature model trained with gradient descent as a solvable model of network training and generalization. Our theory shows how the gap between training and test loss can gradually build up over time due to repeated reuse of data.
  arXiv  Detail & Related papers  (2024-02-02T01:41:38Z)
• A Unified Scheme of ResNet and Softmax [8.556540804058203]
  We provide a theoretical analysis of the regression problem: $| langle exp(Ax) + A x, bf 1_n rangle-1 ( exp(Ax) + Ax ) This regression problem is a unified scheme that combines softmax regression and ResNet, which has never been done before.
  arXiv  Detail & Related papers  (2023-09-23T21:41:01Z)
• Training Multi-Layer Over-Parametrized Neural Network in Subquadratic Time [12.348083977777833]
  We consider the problem of training a multi-layer over-parametrized neural network to minimize the empirical risk induced by a loss function. In this work, we show how to reduce the training cost per iteration.
  arXiv  Detail & Related papers  (2021-12-14T18:13:36Z)
• GradInit: Learning to Initialize Neural Networks for Stable and Efficient Training [59.160154997555956]
  We present GradInit, an automated and architecture method for initializing neural networks. It is based on a simple agnostic; the variance of each network layer is adjusted so that a single step of SGD or Adam results in the smallest possible loss value. It also enables training the original Post-LN Transformer for machine translation without learning rate warmup.
  arXiv  Detail & Related papers  (2021-02-16T11:45:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.