The Weight Norm Sets the Grokking Timescale: A Causal Delay Law

• URL: http://arxiv.org/abs/2606.13753v1
• Date: Thu, 11 Jun 2026 15:36:10 GMT
• Title: The Weight Norm Sets the Grokking Timescale: A Causal Delay Law
• Abstract summary: Grokking is the delayed onset of generalization in neural networks, arising long after they fit the training data.<n>We show that under free training with weight decay, networks grok when the weight norm reaches a value Wc.<n>We instead clamp the norm to a fixed multiple rho of Wc and hold it there, the network still groks, but the delay follows T_grok proportional to exp(alpha rho)
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Grokking is the delayed onset of generalization in neural networks, arising long after they fit the training data. Whether the weight norm causes this delay is disputed: some studies report a critical norm at the transition, others observe grokking with no fixed norm at all. We settle this by intervening on the norm during training rather than only observing it. Under free training with weight decay, networks grok when the weight norm reaches a value Wc that varies little across seeds and learning rates (CV 1 to 2 percent) and grows with the modular base as a power law. When we instead clamp the norm to a fixed multiple rho of Wc and hold it there, the network still groks, but the delay follows T_grok proportional to exp(alpha rho). One exponent, alpha near 7.5, fits this delay across four moduli (R^2 = 0.996). Over the swept ranges the held norm moves the delay by about 19x and the learning rate by only about 2x, and holding the norm above Wc slows grokking rather than preventing it. A final LayerNorm removes the dependence by decoupling weight scale from the network function; without it the exponential law returns. This pinned-norm delay is the exponential counterpart to the logarithmic delay predicted for a freely contracting norm.

Related papers

• Structure-Specific Representational Priors Causally Control the Grokking Delay [0.0]
  Grokking -- generalization long after training-set -- has been accelerated by structure-agnostic interventions.<n>We show it is causally the time to form the right representational structure -- decided at the level of features, not labels.<n>Because the acceleration is gated by a weight-norm side-effect, clamping the norm during training yields a reliable, standalone accelerator.
  arXiv  Detail & Related papers  (2026-07-05T14:32:26Z)
• What Does the Weight Norm Control in Grokking? Logit-Scale Mediation under Cross-Entropy [0.0]
  We ask what the norm actually controls.<n>We slide the grokking delay across its entire norm-induced range under cross-entropy.<n>Across a grid of norms and temperatures the delay collapses onto the logit scale alone (R2 = 0.97), with the norm adding 1-2% beyond it.
  arXiv  Detail & Related papers  (2026-06-16T20:16:18Z)
• Finite-Time Queue Peak Laws in Stochastic Networks: Logarithmic Scaling After Geometric Thresholds [5.115427131586521]
  We study finite-horizon queue peaks in generalized switches.<n>We show that slack reshapes the finite-time peak law for drift-minimizing scheduling policies.
  arXiv  Detail & Related papers  (2026-06-16T17:47:47Z)
• Capacity-Constrained Online Convex Optimization with Delayed Feedback [58.59385794080679]
  We study delayed online convex optimization (OCO) under a hard capacity constraint.<n>We find that for first-order feedback, capacity $C = (log T)$ suffices to recover standard delayed OCO rates up to logarithmic factors.<n>For bandit feedback, the regret rates are modulated by powers of $(1 + _textmax/C)$, where $_textmax$ is the maximum number of pending observations at any time.
  arXiv  Detail & Related papers  (2026-06-10T06:37:06Z)
• Saturating Scaling Laws for Equational Discovery: A Phenomenology of Growth Dynamics in Three Toy Substrates with Two Real-World Replications [0.0]
  We investigate growth dynamics in deterministic equational discovery substrates.<n>Short-range substrate sizes fit a power-law N(t) proportional to tb.<n>A saturating power-law dN/dt = K Nk exp(-mu N) of which the pure power-law is the short-range approximation.
  arXiv  Detail & Related papers  (2026-05-14T21:37:29Z)
• Why Grokking Takes So Long: A First-Principles Theory of Representational Phase Transitions [0.0]
  Grokking is the sudden memorization that appears long after a model has memorized its training data.<n>We present a first-principles theory showing that grokking arises from a norm-driven representational phase transition in regularized training dynamics.
  arXiv  Detail & Related papers  (2026-03-05T17:28:39Z)
• To Grok Grokking: Provable Grokking in Ridge Regression [24.785366757570202]
  We study grokking, the onset of generalization long after overfitting, in a classical ridge regression setting.<n>We show, both theoretically and empirically, that grokking can be amplified or eliminated in a principled manner.<n>Our results suggest grokking is not an inherent failure mode of deep learning, but rather a consequence of specific training conditions.
  arXiv  Detail & Related papers  (2026-01-27T16:52:04Z)
• Seesaw: Accelerating Training by Balancing Learning Rate and Batch Size Scheduling [75.36692892951018]
  Increasing the batch size during training is a promising strategy to accelerate large language model pretraining.<n>This work develops a principled framework for batch-size scheduling.<n>It introduces Seesaw: whenever a standard scheduler would halve the learning rate, Seesaw instead multiplies it by $1/sqrt2$ and doubles the batch size.
  arXiv  Detail & Related papers  (2025-10-16T14:17:38Z)
• VL Norm: Rethink Loss Aggregation in RLVR [23.196933474967224]
  We propose a loss aggregation method tailored to the characteristic of dynamic generation lengths in Reinforcement Learning with Verifiable Rewards (RLVR)<n>By analyzing the effect of varying lengths on policy loss both theoretically and empirically, we reformulate the problem as finding a minimum-variance unbiased estimator.<n>Our proposed VL Norm not only provides an unbiased estimate of the true policy loss but also minimizes gradient variance in theory.
  arXiv  Detail & Related papers  (2025-09-09T09:52:34Z)
• Towards Understanding the Generalizability of Delayed Stochastic Gradient Descent [63.43247232708004]
  A gradient descent performed in an asynchronous manner plays a crucial role in training large-scale machine learning models.<n>Existing generalization error bounds are rather pessimistic and cannot reveal the correlation between asynchronous delays and generalization.<n>Our theoretical results indicate that asynchronous delays reduce the generalization error of the delayed SGD algorithm.
  arXiv  Detail & Related papers  (2023-08-18T10:00:27Z)
• Improved techniques for deterministic l2 robustness [63.34032156196848]
  Training convolutional neural networks (CNNs) with a strict 1-Lipschitz constraint under the $l_2$ norm is useful for adversarial robustness, interpretable gradients and stable training. We introduce a procedure to certify robustness of 1-Lipschitz CNNs by replacing the last linear layer with a 1-hidden layer. We significantly advance the state-of-the-art for standard and provable robust accuracies on CIFAR-10 and CIFAR-100.
  arXiv  Detail & Related papers  (2022-11-15T19:10:12Z)
• On the Overlooked Pitfalls of Weight Decay and How to Mitigate Them: A Gradient-Norm Perspective [90.39123717733334]
  We present the first practical scheduler for weight decay, called the Scheduled Weight Decay (SWD) method. Our experiments also support that SWD indeed mitigates large gradient norms and often significantly outperforms the conventional constant weight decay strategy for Adaptive Moment Estimation (Adam)
  arXiv  Detail & Related papers  (2020-11-23T00:39:49Z)
• A Dynamical Central Limit Theorem for Shallow Neural Networks [48.66103132697071]
  We prove that the fluctuations around the mean limit remain bounded in mean square throughout training. If the mean-field dynamics converges to a measure that interpolates the training data, we prove that the deviation eventually vanishes in the CLT scaling.
  arXiv  Detail & Related papers  (2020-08-21T18:00:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.