Quantifying the Memorization-to-Generalization Transition: Scaling Laws and Phase Structure in Grokking

• URL: http://arxiv.org/abs/2609.10657v1
• Date: Wed, 09 Sep 2026 16:47:03 GMT
• Title: Quantifying the Memorization-to-Generalization Transition: Scaling Laws and Phase Structure in Grokking
• Abstract summary: We map the $0.8-to-generalization boundary across 384 configurations of two-hidden generalizations on modular arithmetic.<n>Data complexity is the dominant driver of regime transition, not model capacity.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Neural networks trained past memorization frequently undergo a delayed transition to generalization, a phenomenon known as grokking. Despite theoretical progress on \emph{why} this transition occurs, the quantitative structure of \emph{when} it occurs in hyperparameter space remains uncharacterized. We map the memorization-to-generalization boundary across 384 configurations of two-hidden-layer MLPs on modular arithmetic, fitting a power-law scaling relation for generalization onset time: $T_{\mathrm{grok}} \propto H^{-0.27}\, D^{-2.04}\, η^{-0.50}\, λ^{-0.64}$ ($R^2 = 0.732$; $0.821$ with interactions). The exponent hierarchy reveals that data complexity ($D^{-2.04}$) is the dominant driver of regime transition, not model capacity ($H^{-0.27}$): doubling data accelerates generalization by ${\sim}4\times$, while doubling width yields only ${\sim}1.2\times$. A sharp phase boundary at weight decay $λ\gtrsim 1.0$ separates grokking from non-grokking configurations, and weight norm trajectories show monotonic compression during the transition, consistent with implicit regularization selecting low-complexity solutions. These results provide a quantitative foundation for predicting and controlling regime transitions in overparameterized networks.

Related papers

• Universal Spin Squeezing Dynamical Phase Transitions across Lattice Geometries, Dimensions, and Microscopic Couplings [0.0]
  Bogoliubov theory recovers the previously identified scaling $a_Z* propto L$ in the long-range interacting regime $ d+2$.<n>This uncovers a previously unrecognized sub-linear regime for short-range interactions.<n>These findings provide a versatile route toward controlling entanglement generation in Rydberg-array, polar molecule, and trapped-ion platforms.
  arXiv  Detail & Related papers  (2026-05-13T18:00:07Z)
• Wavelet Variance Equipartition as a Threshold for World-Model Quality and Quantum Kernel TN-Simulability [0.0]
  We identify the wavelet scaling exponent $$ as a critical diagnostic.<n>We establish $= 1/2$ as a sharp transition boundary for the classical simulability of amplitude-encoded quantum kernels.<n>We prove this variance scales strictly as $Var[X] = (d-2)$.
  arXiv  Detail & Related papers  (2026-05-12T05:41:12Z)
• Weight-Decay Turns Transformer Loss Landscapes Villani: Functional-Analytic Foundations for Optimization and Generalization [2.9582724435853858]
  This paper provides the first rigorous functional-analytic characterization of the standard Transformer objective--cross-entropy loss with $L2$ regularization.<n>We show that the regularized loss $mathcalF$ is infinitely differentiable, grows at least quadratically, has Gaussian-integrable tails, and satisfies the differential growth condition $-mathcalF + tfrac1s|nablamathcalF|2 to infty$ as $|| to
  arXiv  Detail & Related papers  (2026-05-07T17:22:19Z)
• Dimensional Criticality at Grokking Across MLPs and Transformers [2.652953665748039]
  Abrupt transitions between distinct dynamical regimes are a hallmark of complex systems.<n>We introduce textbfTDU--OFC (Thresholded Diffusion Update--Olami-Feder-Christensen), an offline avalanche probe.<n>Across Transformers trained on modular addition and generalizations trained on XOR, we discover a localized crossing of the diffusion baseline $D=1$.
  arXiv  Detail & Related papers  (2026-04-06T13:43:20Z)
• Grokking as a Variance-Limited Phase Transition: Spectral Gating and the Epsilon-Stability Threshold [0.0]
  Generalization requires the textitanisotropic rectification unique to adaptives, which directs noise into the tangent space of the solution.<n>This work analyzes AdamW dynamics on modular arithmetic tasks, revealing a Spectral Gating'' mechanism that regulates the transition from generalization to memorization.
  arXiv  Detail & Related papers  (2026-03-16T16:18:18Z)
• Why Grokking Takes So Long: A First-Principles Theory of Representational Phase Transitions [0.0]
  Grokking is the sudden memorization that appears long after a model has memorized its training data.<n>We present a first-principles theory showing that grokking arises from a norm-driven representational phase transition in regularized training dynamics.
  arXiv  Detail & Related papers  (2026-03-05T17:28:39Z)
• Robust Layerwise Scaling Rules by Proper Weight Decay Tuning [50.11170157029911]
  In modern scale-invariant architectures, training quickly enters an degrading-governed steady state.<n>We introduce a weight-decay scaling rule for AdamW that preserves sublayer gain across widths.<n>Our results extend $mu$P beyond the near-init regime by explicitly controlling the steady-state scales set by parameters.
  arXiv  Detail & Related papers  (2025-10-17T02:58:35Z)
• Emergence and scaling laws in SGD learning of shallow neural networks [64.48316762675141]
  We study the complexity of online gradient descent (SGD) for learning a two-layer neural network with $P$ neurons on isotropic Gaussian data.<n>We provide a precise analysis of SGD dynamics for the training of a student two-layer network to minimize the mean squared error (MSE) objective.
  arXiv  Detail & Related papers  (2025-04-28T16:58:55Z)
• KPZ scaling from the Krylov space [83.88591755871734]
  Recently, a superdiffusion exhibiting the Kardar-Parisi-Zhang scaling in late-time correlators and autocorrelators has been reported. Inspired by these results, we explore the KPZ scaling in correlation functions using their realization in the Krylov operator basis.
  arXiv  Detail & Related papers  (2024-06-04T20:57:59Z)
• Learning with Norm Constrained, Over-parameterized, Two-layer Neural Networks [54.177130905659155]
  Recent studies show that a reproducing kernel Hilbert space (RKHS) is not a suitable space to model functions by neural networks. In this paper, we study a suitable function space for over- parameterized two-layer neural networks with bounded norms.
  arXiv  Detail & Related papers  (2024-04-29T15:04:07Z)
• Improved Algorithm for Adversarial Linear Mixture MDPs with Bandit Feedback and Unknown Transition [71.33787410075577]
  We study reinforcement learning with linear function approximation, unknown transition, and adversarial losses. We propose a new algorithm that attains an $widetildeO(dsqrtHS3K + sqrtHSAK)$ regret with high probability.
  arXiv  Detail & Related papers  (2024-03-07T15:03:50Z)
• Large Stepsize Gradient Descent for Logistic Loss: Non-Monotonicity of the Loss Improves Optimization Efficiency [47.8739414267201]
  We consider gradient descent (GD) with a constant stepsize applied to logistic regression with linearly separable data. We show that GD exits this initial oscillatory phase rapidly -- in $mathcalO(eta)$ steps -- and subsequently achieves an $tildemathcalO (1 / (eta t) )$ convergence rate. Our results imply that, given a budget of $T$ steps, GD can achieve an accelerated loss of $tildemathcalO (1/T2)$ with an aggressive stepsize
  arXiv  Detail & Related papers  (2024-02-24T23:10:28Z)
• Accelerated Gradient Tracking over Time-varying Graphs for Decentralized Optimization [59.65871549878937]
  We prove that the practical single loop accelerated gradient tracking needs $O(fracgamma1-sigma_gamma)2sqrtfracLepsilon)$.<n>Our convergence rates improve significantly over the ones of $O(frac1epsilon5/7)$ and $O(fracLmu)5/7frac1 (1-sigma)1.5logfrac1epsilon)$.
  arXiv  Detail & Related papers  (2021-04-06T15:34:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.