A Stochastic--Geometric Theory of Scaling Laws in Grokking

• URL: http://arxiv.org/abs/2606.30388v2
• Date: Mon, 06 Jul 2026 12:33:31 GMT
• Title: A Stochastic--Geometric Theory of Scaling Laws in Grokking
• Abstract summary: Delayed generalization (iegrokking) refers to the phenomenon in which a neural network fits its training data early in training but only begins to generalize after a prolonged delay.<n>We first theoretically characterize a shell--core topological configuration of the reachable solution space induced by Adam's optimization dynamics with weight-shrinkage regularization.<n>We then analyze the geometry of this topological configuration and the solution transition time at which optimization trajectories escape the memorization manifold.
• Score: 6.109217901547081
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Delayed generalization (\ie~grokking) refers to the phenomenon in which a neural network fits its training data early in training but only begins to generalize after a prolonged delay, often through an abrupt transition. Despite extensive empirical study, its underlying mechanism remains poorly understood. In this work, we first theoretically characterize a shell--core topological configuration of the reachable solution space induced by Adam's optimization dynamics with weight-shrinkage regularization, supported by empirical evidence. This optimization-induced topological configuration gives rise to grokking. In model's parameter space, random initialization solutions concentrate on a thin outer spherical shell, enclosing another spherical shell of memorization solutions, which in turn contains a core corresponding to the generalization solutions. Leveraging stopping-time theory, we then analyze the geometry of this topological configuration and the solution transition time at which optimization trajectories escape the memorization manifold and first reach the boundary of the generalization manifold. Our theoretical analysis derives grokking scaling laws for the learning rate, batch size, and $\ell_2$ regularization coefficient, which are further validated through experiments and shown to recover results from prior literature.

Related papers

• A Theoretical Analysis of Generalization Dynamics in Neural Networks under Gradient Descent with Weight Decay [32.27793372712302]
  We study a class of neural networks trained under the $ell2$ loss by gradient descent (GD) with weight decay.<n>We decompose the population error into data error, optimization error, and prediction variation error, and bound them separately.<n>For the prediction variation error, we propose (local) approximate homogeneity and derive explicit cellwise and layerwise bounds.
  arXiv  Detail & Related papers  (2026-09-07T16:59:16Z)
• Grokking on the Weight-Decay Clock: A Rate Hierarchy from Softly Broken Symmetries [11.824913874212802]
  Delayed generalization, or grokking, remains poorly understood despite extensive empirical study.<n>We identify an exactly solvable late-time relaxation mechanism for grokking in linear models trained with full-batch heavy-ball optimization and weight decay.
  arXiv  Detail & Related papers  (2026-07-27T03:30:44Z)
• Grokking and epoch-wise double descent in quantum neural networks [0.08739101659113156]
  We report the empirical observation of both the grokking transition and epoch-wise double descent in a two-qubit quantum neural network (QNN)<n>We identify a generalization decay in late-stage training, where the test error increases significantly despite a stagnant training loss.<n>Our analysis reveals that this decay correlates with an unconstrained increase of the weight-norm, drifting away from sparse, phase-aligned harmonic solutions.
  arXiv  Detail & Related papers  (2026-07-09T10:55:16Z)
• On Higher-Order Geometric Refinements of Classical Covariance Asymptotics: An Approach via Intrinsic and Extrinsic Information Geometry [0.0]
  In curved models, including mixtures, curved exponential families, latent-variable models, and manifold-root parameter spaces, finite-sample behavior can deviate systematically from predictions.<n>We develop a coordinate-invariant, curvature-aware refinement by viewing a regular parametric family as a ((,g)) with Fisher-constrained-Rao metric.<n>We describe the resolved metric, the role of the real log canonical threshold in learning rates and posterior mean-squared error, and a curvature-based covariance expansion on the resolved space that recovers the regular theory as a special case.
  arXiv  Detail & Related papers  (2026-04-14T13:40:13Z)
• Beyond Perturbation Theory: A Resolvent-Based Framework for Strongly Correlated Many-Body Systems [1.4824891788575416]
  Traditional perturbation theory fails in many-body systems with exponentially small energy gaps and strong interactions.<n>This work presents an alternative methodological framework built on two core principles.<n>It is applicable to the quantitative analysis of global properties in nonintegrable many-body systems.
  arXiv  Detail & Related papers  (2026-04-01T08:11:58Z)
• On Multi-Step Theorem Prediction via Non-Parametric Structural Priors [50.16583672681106]
  In this work, we explore training-free theorem prediction through the lens of in-context learning (ICL)<n>We propose Theorem Precedence Graphs, which encode temporal dependencies from historical solution traces as directed graphs, and impose explicit topological constraints that effectively prune the search space during inference.<n>Experiments on the FormalGeo7k benchmark show that our method achieves 89.29% accuracy, substantially outperforming ICL baselines and matching state-of-the-art supervised models.
  arXiv  Detail & Related papers  (2026-03-05T06:08:50Z)
• The Procrustean Bed of Time Series: The Optimization Bias of Point-wise Loss [53.542743390809356]
  This paper aims to provide a first-principles analysis of the Expectation of Optimization Bias (EOB)<n>Our analysis reveals a fundamental paradigm paradox: the more deterministic and structured the time series, the more severe the bias by point-wise loss function.<n>We present a concrete solution that simultaneously achieves both principles via DFT or DWT.
  arXiv  Detail & Related papers  (2025-12-21T06:08:22Z)
• The Geometry of Grokking: Norm Minimization on the Zero-Loss Manifold [5.076419064097734]
  We argue that post-memorization learning can be understood through the lens of constrained optimization.<n>We show that gradient descent effectively minimizes the weight norm on the zero-loss manifold.<n>Experiments confirm that simulating the training process using our predicted gradients reproduces both the delayed generalization and representation learning characteristic of grokking.
  arXiv  Detail & Related papers  (2025-11-02T18:44:42Z)
• Beyond Progress Measures: Theoretical Insights into the Mechanism of Grokking [50.465604300990904]
  Grokking refers to the abrupt improvement in test accuracy after extended overfitting.<n>In this work, we investigate the grokking mechanism underlying the Transformer in the task of prime number operations.
  arXiv  Detail & Related papers  (2025-04-04T04:42:38Z)
• On the Dynamics Under the Unhinged Loss and Beyond [104.49565602940699]
  We introduce the unhinged loss, a concise loss function, that offers more mathematical opportunities to analyze closed-form dynamics. The unhinged loss allows for considering more practical techniques, such as time-vary learning rates and feature normalization.
  arXiv  Detail & Related papers  (2023-12-13T02:11:07Z)
• Stochastic parameter optimization analysis of dynamical quantum critical phenomena in long-range transverse-field Ising chain [0.0]
  We explore the quantum phase transition of the one-dimensional long-range transverse-field Ising model. In our simulations, the simulator automatically determines the parameters to sample from, even without prior knowledge of the critical point and universality class. We successfully obtained numerical evidence supporting $sigma = 7/4$ as the universality boundary between the latter two.
  arXiv  Detail & Related papers  (2023-05-23T14:46:16Z)
• Understanding the Generalization Ability of Deep Learning Algorithms: A Kernelized Renyi's Entropy Perspective [11.255943520955764]
  We propose a novel information theoretical measure: kernelized Renyi's entropy. We establish the generalization error bounds for gradient/Langevin descent (SGD/SGLD) learning algorithms under kernelized Renyi's entropy. We show that our information-theoretical bounds depend on the statistics of the gradients, and are rigorously tighter than the current state-of-the-art (SOTA) results.
  arXiv  Detail & Related papers  (2023-05-02T01:17:15Z)
• Stability and Generalization Analysis of Gradient Methods for Shallow Neural Networks [59.142826407441106]
  We study the generalization behavior of shallow neural networks (SNNs) by leveraging the concept of algorithmic stability. We consider gradient descent (GD) and gradient descent (SGD) to train SNNs, for both of which we develop consistent excess bounds.
  arXiv  Detail & Related papers  (2022-09-19T18:48:00Z)
• Fractal Structure and Generalization Properties of Stochastic Optimization Algorithms [71.62575565990502]
  We prove that the generalization error of an optimization algorithm can be bounded on the complexity' of the fractal structure that underlies its generalization measure. We further specialize our results to specific problems (e.g., linear/logistic regression, one hidden/layered neural networks) and algorithms.
  arXiv  Detail & Related papers  (2021-06-09T08:05:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.