Deciphering Two Training Clocks in Grokking via Deep Linear Network Theory with Conditional ReLU Reduction

• URL: http://arxiv.org/abs/2606.05863v1
• Date: Thu, 04 Jun 2026 08:39:04 GMT
• Title: Deciphering Two Training Clocks in Grokking via Deep Linear Network Theory with Conditional ReLU Reduction
• Abstract summary: Grokking suggests that fitting training data and learning a simple underlying rule may occur on different time scales.<n>We formalize this phenomenon by separating the fast decay of the classification loss from the slower simplification of the learned representation.<n>We show that a post-margin gap-growth or one-step-contraction condition reduces the cross-entropy loss to level epsilon on a logarithmic time scale.
• Score: 18.72807692009739
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Grokking suggests that fitting the training data and learning a simple underlying rule may occur on different time scales. We formalize this phenomenon by separating the fast decay of the classification loss from the slower simplification of the learned representation, and we call the resulting pair of stopping times two training clocks. For deep linear networks, we show that a post-margin gap-growth or one-step tail-contraction condition reduces the cross-entropy loss to level epsilon on a logarithmic time scale. In contrast, when layerwise weight decay is present, the induced regularization on the end-to-end map can be expressed as a Schatten-type penalty; under a sharp late-time Kurdyka-Lojasiewicz tail, this structural energy closes on a polynomial time scale. The two clocks, therefore, separate fitting from representation simplification. We then explain how the same mechanism can appear in ReLU MLPs. In regions where the activation patterns on the training set remain fixed, the network reduces to a linear model in the active coordinates. In a two-layer ReLU embedding model, chain-rule estimates further show that the classifier head can receive larger effective gradients than the embedding block under controlled downstream norms. This supports a two-stage mechanism in which the classifier fits first, while the representation continues to simplify later. We use modular addition as the main experimental setting. The deep linear theory provides the rigorous core of the analysis. But the ReLU results are formulated as conditional reductions that account for empirical behavior without claiming a global proof for nonlinear training dynamics.

Related papers

• Implicit Bias of SGD in Multivariate ReLU Networks: Effective Width Collapse [16.78534503577613]
  We show that the input weights and biases align along finitely many directions, leading to an effective width collapse.<n>We establish a non-redundancy property of the learned representation by proving that each learned direction induces a unique ternary activation pattern on the training data.
  arXiv  Detail & Related papers  (2026-07-03T22:03:13Z)
• Monitoring Neural Training with Topology: A Footprint-Predictable Collapse Index [51.56484100374058]
  We propose an online, topology-aware monitor for evolving neural representations that couples Modular Morse Homology Maintenance (MMHM) with a composite Collapse Index (CI)<n>Instead of rebuilding complexes each epoch, we apply sparse edits at a fixed scale and maintain a discrete Morse matching, yielding fast, incremental updates.
  arXiv  Detail & Related papers  (2026-04-28T19:21:21Z)
• Random Matrix Theory of Early-Stopped Gradient Flow: A Transient BBP Scenario [1.2744523252873352]
  We show that learning occurs when an isolated eigenvalue separates from a noisy bulk, before disappearing in the overfitting regime.<n>We map the corresponding phase diagrams and validate the theory against finite-size simulations.
  arXiv  Detail & Related papers  (2026-04-20T16:05:58Z)
• The Geometry of Grokking: Norm Minimization on the Zero-Loss Manifold [5.076419064097734]
  We argue that post-memorization learning can be understood through the lens of constrained optimization.<n>We show that gradient descent effectively minimizes the weight norm on the zero-loss manifold.<n>Experiments confirm that simulating the training process using our predicted gradients reproduces both the delayed generalization and representation learning characteristic of grokking.
  arXiv  Detail & Related papers  (2025-11-02T18:44:42Z)
• A dynamic view of some anomalous phenomena in SGD [0.0]
  It has been observed that over-parametrized neural networks exhibit a double descent' phenomenon.<n>Another anomalous phenomenon is that of itgrokking wherein two regimes of descent are interrupted by a third regime wherein the mean loss remains almost constant.<n>This note presents a plausible explanation for these and related phenomena by using the theory of two time scale approximation, applied to the continuous time limit dynamics.
  arXiv  Detail & Related papers  (2025-05-03T09:01:10Z)
• Intensity Profile Projection: A Framework for Continuous-Time Representation Learning for Dynamic Networks [50.2033914945157]
  We present a representation learning framework, Intensity Profile Projection, for continuous-time dynamic network data. The framework consists of three stages: estimating pairwise intensity functions, learning a projection which minimises a notion of intensity reconstruction error. Moreoever, we develop estimation theory providing tight control on the error of any estimated trajectory, indicating that the representations could even be used in quite noise-sensitive follow-on analyses.
  arXiv  Detail & Related papers  (2023-06-09T15:38:25Z)
• Training invariances and the low-rank phenomenon: beyond linear networks [44.02161831977037]
  We show that when one trains a deep linear network with logistic or exponential loss on linearly separable data, the weights converge to rank-$1$ matrices. This is the first time a low-rank phenomenon is proven rigorously for nonlinear ReLU-activated feedforward networks. Our proof relies on a specific decomposition of the network into a multilinear function and another ReLU network whose weights are constant under a certain parameter directional convergence.
  arXiv  Detail & Related papers  (2022-01-28T07:31:19Z)
• Multi-scale Feature Learning Dynamics: Insights for Double Descent [71.91871020059857]
  We study the phenomenon of "double descent" of the generalization error. We find that double descent can be attributed to distinct features being learned at different scales.
  arXiv  Detail & Related papers  (2021-12-06T18:17:08Z)
• A Convergence Theory Towards Practical Over-parameterized Deep Neural Networks [56.084798078072396]
  We take a step towards closing the gap between theory and practice by significantly improving the known theoretical bounds on both the network width and the convergence time. We show that convergence to a global minimum is guaranteed for networks with quadratic widths in the sample size and linear in their depth at a time logarithmic in both. Our analysis and convergence bounds are derived via the construction of a surrogate network with fixed activation patterns that can be transformed at any time to an equivalent ReLU network of a reasonable size.
  arXiv  Detail & Related papers  (2021-01-12T00:40:45Z)
• Early Stopping in Deep Networks: Double Descent and How to Eliminate it [30.61588337557343]
  We show that epoch-wise double descent arises because different parts of the network are learned at different epochs. We study two standard convolutional networks empirically and show that eliminating epoch-wise double descent through adjusting stepsizes of different layers improves the early stopping performance significantly.
  arXiv  Detail & Related papers  (2020-07-20T13:43:33Z)
• Neural Networks are Convex Regularizers: Exact Polynomial-time Convex Optimization Formulations for Two-layer Networks [70.15611146583068]
  We develop exact representations of training two-layer neural networks with rectified linear units (ReLUs) Our theory utilizes semi-infinite duality and minimum norm regularization.
  arXiv  Detail & Related papers  (2020-02-24T21:32:41Z)
• Kernel and Rich Regimes in Overparametrized Models [69.40899443842443]
  We show that gradient descent on overparametrized multilayer networks can induce rich implicit biases that are not RKHS norms. We also demonstrate this transition empirically for more complex matrix factorization models and multilayer non-linear networks.
  arXiv  Detail & Related papers  (2020-02-20T15:43:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.