What Does the Weight Norm Control in Grokking? Logit-Scale Mediation under Cross-Entropy

• URL: http://arxiv.org/abs/2606.18465v1
• Date: Tue, 16 Jun 2026 20:16:18 GMT
• Title: What Does the Weight Norm Control in Grokking? Logit-Scale Mediation under Cross-Entropy
• Abstract summary: We ask what the norm actually controls.<n>We slide the grokking delay across its entire norm-induced range under cross-entropy.<n>Across a grid of norms and temperatures the delay collapses onto the logit scale alone (R2 = 0.97), with the norm adding 1-2% beyond it.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Grokking, the delayed jump from memorization to generalization, is usually tied to the weight norm: a smaller norm generalizes sooner. We ask what the norm actually controls. Holding the weight norm fixed by clamping and varying only an output temperature, we slide the grokking delay across its entire norm-induced range under cross-entropy; matching the effective logit scale back to baseline recovers about 85% of the delay at two moduli. Across a grid of norms and temperatures the delay collapses onto the logit scale alone (R2 = 0.97), with the norm adding 1-2% beyond it. The effect is loss-dependent: under mean-squared error the logit scale is pinned and the norm acts through a different route. A memorization control, a float64 softmax-collapse audit, and a no-LayerNorm transformer point to the same channel. Forking arms from one identical state, the delay follows the held norm value and not the clamp operation, which closes a rescaling-artifact concern. The proximal variable is the logit scale and the softmax saturation it drives; the weight norm is only an upstream handle. All numbers, tables, and figures reproduce from released code and data.

Related papers

• Feature Priming in Online Linear Regression: Sparse-Regret Lower Bounds and a Tight Univariate Rate [47.36630149538994]
  In high-dimensional online prediction, the best predictor may depend on only a few features, so regret should scale with sparsity rather than the ambient dimension.<n>Warmuth and Amid asked at COLT 2023 whether any of three such rules admits a competitive online regret guarantee.<n>Our analysis identifies a common obstruction: cheap nuisance causes the refit to underweight the truly predictive coordinate.
  arXiv  Detail & Related papers  (2026-08-18T09:32:03Z)
• DeepLoop: Depth Scaling for Looped Transformers [91.7922038545625]
  Looped Transformers scale sequential computation by applying a compact stack of physical blocks for multiple rounds.<n>We formalize this tied-depth effect through a first-order perturbation bound controlled by a visit-alignment coefficient.<n>The resulting method, textbfDeepLoop, keeps the Post-LN DeepNorm architecture and sets $= (2N)1/2$ and $=(8N)-1/2$ for unrolled depth.
  arXiv  Detail & Related papers  (2026-07-15T06:37:05Z)
• At-Grok Is Not Converged:A Measurement-Validity Audit for Grokking Representation Metrics [0.0]
  We show that compression lags the accuracy transition by an amount on the time-to-grok, at least 10,000 steps, rather than coinciding with it.<n>We package this as an audit that separates onset from compression flags, censoring, excludes cells that never fully generalize, and checks that the reference floor has plateaued.
  arXiv  Detail & Related papers  (2026-07-07T14:44:18Z)
• Dense Supervision Is Not Enough: The Readout Blind Spot in Looped Language Models [5.903857251549266]
  We show that dense per-loop cross-entropy controls the variables exposed by the readout, not every variable active in the recurrent transition.<n>In 44M and 129M looped transformers without inter-loop normalization, per-loop cross-entropy through RMSNorm readouts still drives final hidden-state norms into the thousands or tens of thousands.
  arXiv  Detail & Related papers  (2026-06-12T13:23:18Z)
• The Weight Norm Sets the Grokking Timescale: A Causal Delay Law [0.0]
  Grokking is the delayed onset of generalization in neural networks, arising long after they fit the training data.<n>We show that under free training with weight decay, networks grok when the weight norm reaches a value Wc.<n>We instead clamp the norm to a fixed multiple rho of Wc and hold it there, the network still groks, but the delay follows T_grok proportional to exp(alpha rho)
  arXiv  Detail & Related papers  (2026-06-11T15:36:10Z)
• Predictive Autoscaling for Node.js on Kubernetes: Lower Latency, Right-Sized Capacity [0.0]
  We propose a predictive scaling algorithm that forecasts where load will be by the time new capacity is ready.<n>Under steady ramp, median latency is 26ms, compared to 154ms for KEDA and 522ms for HPA.
  arXiv  Detail & Related papers  (2026-04-21T17:25:39Z)
• Gated Removal of Normalization in Transformers Enables Stable Training and Efficient Inference [2.1665689529884697]
  We introduce TaperNorm, a drop-in replacement for RMSNorm/LayerNorm that behaves exactly like the standard normalizer early in training.<n>A single global gate is held at $g=1$ during gate warmup, used to calibrate the scaling branch via EMAs, and then cosine-decayed to $g=0$.<n>On an efficiency microbenchmark, folding internal scalings yields up to $1.22times$ higher throughput in last-token logits mode.
  arXiv  Detail & Related papers  (2026-02-11T01:40:34Z)
• Late-Stage Generalization Collapse in Grokking: Detecting anti-grokking with Weightwatcher [1.6615337656760856]
  emphMemorization in neural networks lacks a precise operational definition and is often inferred from the grokking regime.<n>We identify a previously unreported third phase of grokking in this training regime: emphanti-grokking, a late-stage collapse of generalization.
  arXiv  Detail & Related papers  (2026-02-02T22:09:14Z)
• Gated KalmaNet: A Fading Memory Layer Through Test-Time Ridge Regression [53.48692193399171]
  Gated KalmaNet (GKA) is a layer that reduces the gap by accounting for the full past when predicting the next token.<n>We solve an online ridge regression problem at test time, with constant memory and linear compute cost in the sequence length.<n>On long-context, GKA excels at real-world RAG and LongQA tasks up to 128k tokens, achieving more than $10$% relative improvement over other fading memory baselines.
  arXiv  Detail & Related papers  (2025-11-26T03:26:37Z)
• INC: An Indirect Neural Corrector for Auto-Regressive Hybrid PDE Solvers [61.84396402100827]
  We propose the Indirect Neural Corrector ($mathrmINC$), which integrates learned corrections into the governing equations.<n>$mathrmINC$ reduces the error amplification on the order of $t-1 + L$, where $t$ is the timestep and $L$ the Lipschitz constant.<n>We test $mathrmINC$ in extensive benchmarks, covering numerous differentiable solvers, neural backbones, and test cases ranging from a 1D chaotic system to 3D turbulence.
  arXiv  Detail & Related papers  (2025-11-16T20:14:28Z)
• Robust Layerwise Scaling Rules by Proper Weight Decay Tuning [50.11170157029911]
  In modern scale-invariant architectures, training quickly enters an degrading-governed steady state.<n>We introduce a weight-decay scaling rule for AdamW that preserves sublayer gain across widths.<n>Our results extend $mu$P beyond the near-init regime by explicitly controlling the steady-state scales set by parameters.
  arXiv  Detail & Related papers  (2025-10-17T02:58:35Z)
• Robust Training of Neural Networks using Scale Invariant Architectures [70.67803417918854]
  In contrast to SGD, adaptive gradient methods like Adam allow robust training of modern deep networks. We show that this general approach is robust to rescaling of parameter and loss. We design a scale invariant version of BERT, called SIBERT, which when trained simply by vanilla SGD achieves performance comparable to BERT trained by adaptive methods like Adam.
  arXiv  Detail & Related papers  (2022-02-02T11:58:56Z)
• On the Overlooked Pitfalls of Weight Decay and How to Mitigate Them: A Gradient-Norm Perspective [90.39123717733334]
  We present the first practical scheduler for weight decay, called the Scheduled Weight Decay (SWD) method. Our experiments also support that SWD indeed mitigates large gradient norms and often significantly outperforms the conventional constant weight decay strategy for Adaptive Moment Estimation (Adam)
  arXiv  Detail & Related papers  (2020-11-23T00:39:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.