Weight Decay Regimes in Grokking Transformers: Cheap Online Diagnostics

• URL: http://arxiv.org/abs/2605.20441v1
• Date: Tue, 19 May 2026 19:48:40 GMT
• Title: Weight Decay Regimes in Grokking Transformers: Cheap Online Diagnostics
• Abstract summary: Transformers trained on modular arithmetic exhibit sharp transitions between memorization, generalization, and collapse.<n>We show that weight decay acts as a scalar empirical control parameter for these regimes.<n>We introduce online diagnostics, mean pairwise attentionhead cosine similarity and entropy standard deviation, that track training dynamics.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Transformers trained on modular arithmetic exhibit sharp transitions between memorization, generalization, and collapse. We show that weight decay acts as a scalar empirical control parameter for these regimes, and introduce two cheap online diagnostics, mean pairwise attention-head cosine similarity and entropy standard deviation, that track training dynamics from attention activations alone and complement loss-landscape diagnostics at lower compute cost. Across eleven experimental conditions and three model scales (0.82M to 85M parameters), the weight-decay axis separates memorization, developmental grokking, and collapse. A near-transition logistic fit localizes the memorization-to-developmental boundary at $λ_c=0.0158$ (95% CI [0.0109, 0.0200], N=210); a power-law fit gives an empirical exponent $ν=0.757$ (CI [0.725, 0.799]). Reference exponents $ν=1/2$ and 3D Ising $ν\approx 0.63$ lie outside this empirical CI under our four-bin grid, so we report $ν$ as empirical and defer universality-class identification to denser finite-size-scaling work. A horizon-matched multi-task replication (n=280, four modular operations) preserves the weight-decay control pattern; a paired attention-head re-initialization experiment at $λ=0.05$ changes Phase-2 amplitude (Cohen's $d=-1.190$, n=10, $p_t=4.5 \times 10^{-3}$), while matched weight-norm clipping does not. Three cross-architecture probes (4L MLP, 4L LSTM, and 4L Mamba; each n=70) replicate the weight-decay-controlled transition with architecture-specific $λ_c$ values. Main diagnostic claims are scoped to modular arithmetic in small transformer attention models; the non-attention experiments are scope probes, and architecture-wide, language-model, and universality-class claims are out of scope.

Related papers

• TextNCA: Neural Cellular Automata for Language Modeling via Hierarchical Local Attention [45.024660338672675]
  We define textscTextNCA, a 1D causal windowed-attention realisation of the Neural Cellular Automaton primitive.<n>We study a hierarchical variant that cascades three stages with windows $w in 8, 32, 128$ and $T_s$ shared-weight iterations per stage, all on WikiText-103 at roughly 30M parameters and 60k training steps.
  arXiv  Detail & Related papers  (2026-08-03T10:47:05Z)
• The Intruder Threshold: A Spectral Law for LoRA Fine-Tuning [3.860898721527665]
  LoRA fine-tuning can create intruder dimensions that drive catastrophic forgetting.<n>No theory has predicted, layer by layer on measured spectra, when they appear.<n>Full fine-tuning disperses its update far below the threshold of every layer.<n>Norm-matched interventions confirm that threshold-crossing layers, rather than update magnitude, carry the forgetting.
  arXiv  Detail & Related papers  (2026-07-26T15:17:41Z)
• Derivative Informed Learning of Exchange-Correlation Functionals [46.44973075720396]
  Machine-learned (ML) exchange-correlation (XC) functionals aim to replace human-designed density functional approximations.<n>We study a hybrid-distillation setting in which $mathcalO(N4)$-scaling ML-XC functionals are trained to reproduce B3LYP/def2-SVP targets.<n>We introduce Derivative Informed XC-Loss (DI-Loss), a loss that incorporates additional information from the reference hybrid functional.
  arXiv  Detail & Related papers  (2026-06-02T23:07:54Z)
• Transformers Provably Learn to Internalize Chain-of-Thought [65.41010769606844]
  Implicit Chain-of-Thought (ICoT) trains models to internalize intermediate steps within their hidden states.<n>Log-ICoT removes thinking tokens one at a time, reducing the number of stages from linear in $k$ to logarithmic.<n>Experiments on multi-layer transformers confirm the theory and visualize how reasoning is progressively absorbed into deeper layers.
  arXiv  Detail & Related papers  (2026-05-27T15:17:06Z)
• How to Scale Mixture-of-Experts: From muP to the Maximally Scale-Stable Parameterization [45.69980208532521]
  We take a principled step toward resolving this gap by analyzing three different scaling regimes.<n>For each regime, we develop a novel Dynamical Mean Field Theory (DMFT) description of the limiting training dynamics of MoEs.<n>We show that the resulting $$P prescription does not reliably induce monotonic improvement with scale or robust learning-rate transfer.
  arXiv  Detail & Related papers  (2026-05-13T23:32:00Z)
• Lying Is Just a Phase: The Hidden Alignment Transition in Language Model Scaling [0.0]
  We measure the coupling between reasoning and truthfulness across 63 base models from 16 families.<n>We find a regime change invisible to loss curves below a family-dependent critical scale $N_c$, capabilities anticorrelate; above it, they cooperate.
  arXiv  Detail & Related papers  (2026-05-13T03:14:09Z)
• Wavelet Variance Equipartition as a Threshold for World-Model Quality and Quantum Kernel TN-Simulability [0.0]
  We identify the wavelet scaling exponent $$ as a critical diagnostic.<n>We establish $= 1/2$ as a sharp transition boundary for the classical simulability of amplitude-encoded quantum kernels.<n>We prove this variance scales strictly as $Var[X] = (d-2)$.
  arXiv  Detail & Related papers  (2026-05-12T05:41:12Z)
• What Do EEG Foundation Models Capture from Human Brain Signals? [64.48249643001402]
  Modern EEG foundation models learn directly from raw signals via self-supervised pretraining.<n>We decompose into three sub-questions: emphwhat does the model learn, emphwhat does the model use, and emphhow much can be explained<n>The audit covers three foundation models (CSBrain, CBraMod, LaBraM), five clinical tasks (MDD, Stress, ISRUC-Sleep, TUSL, Siena), and a 6-family 63-feature lexicon.
  arXiv  Detail & Related papers  (2026-05-12T01:57:53Z)
• The Geometry of Forgetting: Temporal Knowledge Drift as an Independent Axis in LLM Representations [50.43168858368539]
  Large language models confidently produce outdated answers, and no existing method can detect them.<n>We show this is not an engineering failure but a structural one: temporal drift is encoded as a direction in the residual stream geometrically to both correctness and uncertainty.
  arXiv  Detail & Related papers  (2026-05-09T22:27:31Z)
• When Self-Reference Fails to Close: Matrix-Level Dynamics in Large Language Models [0.0]
  Self-referential statements and meta-cognitive prompts are more stable than paradoxical self-reference on key collapse-related metrics.<n>We propose, as a conjecture, that NCTR forces finite-depth transformers toward dynamical regimes where these problems concentrate.
  arXiv  Detail & Related papers  (2026-04-13T23:23:02Z)
• Spectral Edge Dynamics of Training Trajectories: Signal--Noise Geometry Across Scales [0.0]
  We show that transformer training trajectories evolve within only a few coherent directions.<n>In companion work, the same spectral geometry provides early-warning signals of grokking.
  arXiv  Detail & Related papers  (2026-03-14T04:46:05Z)
• First-Principles Optical Descriptors and Hybrid Classical-Quantum Classification of Er-Doped CaF$_2$ [3.039491211255488]
  We present a classical-quantum machine learning framework for discriminating pristine CaF$ from Er-doped CaF$$ using first-principles optical descriptors.<n>From 1,589 energy-resolved points per system, physically interpretable descriptors including transition energy $E$, extinction coefficient $$, and absorption coefficient $$ were extracted.
  arXiv  Detail & Related papers  (2026-01-31T05:43:59Z)
• Robust Layerwise Scaling Rules by Proper Weight Decay Tuning [50.11170157029911]
  In modern scale-invariant architectures, training quickly enters an degrading-governed steady state.<n>We introduce a weight-decay scaling rule for AdamW that preserves sublayer gain across widths.<n>Our results extend $mu$P beyond the near-init regime by explicitly controlling the steady-state scales set by parameters.
  arXiv  Detail & Related papers  (2025-10-17T02:58:35Z)
• Tensor Decomposition Networks for Fast Machine Learning Interatomic Potential Computations [48.46721044282335]
  tensor decomposition networks (TDNs) achieve competitive performance with dramatic speedup in computations.<n>We evaluate TDNs on PubChemQCR, a newly curated molecular relaxation dataset containing 105 million DFT-calculated snapshots.<n>Results show that TDNs achieve competitive performance with dramatic speedup in computations.
  arXiv  Detail & Related papers  (2025-07-01T18:46:27Z)
• Minimum optical depth multiport interferometers for approximating arbitrary unitary operations and pure states [37.69303106863453]
  We address the problem of, using multiport interferometers, approximating with a given infidelity any pure state preparation and any unitary operation.<n>By means of numerical calculations, we show that pure states, in any dimension $d$, can be prepared with infidelity.
  arXiv  Detail & Related papers  (2020-02-04T15:40:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.