Using physics-inspired Singular Learning Theory to understand grokking & other phase transitions in modern neural networks

• URL: http://arxiv.org/abs/2512.00686v3
• Date: Wed, 03 Dec 2025 22:14:33 GMT
• Title: Using physics-inspired Singular Learning Theory to understand grokking & other phase transitions in modern neural networks
• Abstract summary: We study Singular learning theory (SLT) in toy settings relevant to interpretability and phase transitions.<n>Our paper illustrates the many merits of SLT for understanding neural network phase transitions, and poses open research questions for the field.
• Score: 0.913755431537592
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Classical statistical inference and learning theory often fail to explain the success of modern neural networks. A key reason is that these models are non-identifiable (singular), violating core assumptions behind PAC bounds and asymptotic normality. Singular learning theory (SLT), a physics-inspired framework grounded in algebraic geometry, has gained popularity for its ability to close this theory-practice gap. In this paper, we empirically study SLT in toy settings relevant to interpretability and phase transitions. First, we understand the SLT free energy $\mathcal{F}_n$ by testing an Arrhenius-style rate hypothesis using both a grokking modulo-arithmetic model and Anthropic's Toy Models of Superposition. Second, we understand the local learning coefficient $λ_α$ by measuring how it scales with problem difficulty across several controlled network families (polynomial regressors, low-rank linear networks, and low-rank autoencoders). Our experiments recover known scaling laws while others yield meaningful deviations from theoretical expectations. Overall, our paper illustrates the many merits of SLT for understanding neural network phase transitions, and poses open research questions for the field.

Related papers

• Precise gradient descent training dynamics for finite-width multi-layer neural networks [8.057006406834466]
  We provide the first precise distributional characterization of gradient descent iterates for general multi-layer neural networks.<n>Our non-asymptotic state evolution theory captures Gaussian fluctuations in first-layer weights and concentration in deeper-layer weights.
  arXiv  Detail & Related papers  (2025-05-08T02:19:39Z)
• A Near Complete Nonasymptotic Generalization Theory For Multilayer Neural Networks: Beyond the Bias-Variance Tradeoff [57.25901375384457]
  We propose a nonasymptotic generalization theory for multilayer neural networks with arbitrary Lipschitz activations and general Lipschitz loss functions.<n>In particular, it doens't require the boundness of loss function, as commonly assumed in the literature.<n>We show the near minimax optimality of our theory for multilayer ReLU networks for regression problems.
  arXiv  Detail & Related papers  (2025-03-03T23:34:12Z)
• Feature Learning Beyond the Edge of Stability [8.430481660019451]
  We propose a homogeneous multilayer perceptron parameterization with hidden layer width pattern and analyze its training dynamics under gradient descent.<n>We obtain formulas for the first three Taylor coefficients of the minibatch loss during training that illuminate the connection between sharpness and feature learning.
  arXiv  Detail & Related papers  (2025-02-18T18:23:33Z)
• Deep learning lattice gauge theories [0.0]
  We use neural network quantum states to accurately compute the ground state of lattice gauge theories in $2+1$ dimensions.<n>Our findings suggest that neural network quantum states are a promising method for precise studies of lattice gauge theory.
  arXiv  Detail & Related papers  (2024-05-23T17:46:49Z)
• On Learning Latent Models with Multi-Instance Weak Supervision [57.18649648182171]
  We consider a weakly supervised learning scenario where the supervision signal is generated by a transition function $sigma$ labels associated with multiple input instances. Our problem is met in different fields, including latent structural learning and neuro-symbolic integration.
  arXiv  Detail & Related papers  (2023-06-23T22:05:08Z)
• The Quantization Model of Neural Scaling [19.057931064238584]
  We propose the Quantization Model of neural scaling laws, explaining both the observed power law dropoff of loss with model and data size, and the sudden emergence of new capabilities with scale. We show that when quanta are learned in order of decreasing use frequency, then a power law in use explains observed power law scaling of loss.
  arXiv  Detail & Related papers  (2023-03-23T17:58:43Z)
• A Unified Algebraic Perspective on Lipschitz Neural Networks [88.14073994459586]
  This paper introduces a novel perspective unifying various types of 1-Lipschitz neural networks. We show that many existing techniques can be derived and generalized via finding analytical solutions of a common semidefinite programming (SDP) condition. Our approach, called SDP-based Lipschitz Layers (SLL), allows us to design non-trivial yet efficient generalization of convex potential layers.
  arXiv  Detail & Related papers  (2023-03-06T14:31:09Z)
• Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limit [36.17720004582283]
  This work conducts such an exploration through the lens of learning $k$-sparse parities of $n$ bits. We find that neural networks exhibit surprising phase transitions when scaling up dataset size and running time.
  arXiv  Detail & Related papers  (2022-07-18T17:55:05Z)
• Generalization Through The Lens Of Leave-One-Out Error [22.188535244056016]
  We show that the leave-one-out error provides a tractable way to estimate the generalization ability of deep neural networks in the kernel regime. Our work therefore demonstrates that the leave-one-out error provides a tractable way to estimate the generalization ability of deep neural networks in the kernel regime.
  arXiv  Detail & Related papers  (2022-03-07T14:56:00Z)
• Multi-scale Feature Learning Dynamics: Insights for Double Descent [71.91871020059857]
  We study the phenomenon of "double descent" of the generalization error. We find that double descent can be attributed to distinct features being learned at different scales.
  arXiv  Detail & Related papers  (2021-12-06T18:17:08Z)
• Why Lottery Ticket Wins? A Theoretical Perspective of Sample Complexity on Pruned Neural Networks [79.74580058178594]
  We analyze the performance of training a pruned neural network by analyzing the geometric structure of the objective function. We show that the convex region near a desirable model with guaranteed generalization enlarges as the neural network model is pruned.
  arXiv  Detail & Related papers  (2021-10-12T01:11:07Z)
• Towards an Understanding of Benign Overfitting in Neural Networks [104.2956323934544]
  Modern machine learning models often employ a huge number of parameters and are typically optimized to have zero training loss. We examine how these benign overfitting phenomena occur in a two-layer neural network setting. We show that it is possible for the two-layer ReLU network interpolator to achieve a near minimax-optimal learning rate.
  arXiv  Detail & Related papers  (2021-06-06T19:08:53Z)
• A Neural Network Perturbation Theory Based on the Born Series [0.0]
  Taylor coefficients of deep neural networks (DNNs) still appear mainly in the light of interpretability studies. This gap motivates a general formulation of neural network (NN) Taylor expansions. We show that NNs adapt their derivatives mainly to the leading order of the target function's Taylor expansion.
  arXiv  Detail & Related papers  (2020-09-07T15:54:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.