The Lifecycle of the Spectral Edge: From Gradient Learning to Weight-Decay Compression

• URL: http://arxiv.org/abs/2604.07380v1
• Date: Wed, 08 Apr 2026 01:57:04 GMT
• Title: The Lifecycle of the Spectral Edge: From Gradient Learning to Weight-Decay Compression
• Abstract summary: We decompose the spectral edge into its gradient and weight-decay components during grokking in two sequence tasks.<n>We find a sharp two-phase lifecycle: before grokking the edge is universality-driven and functionally active; at grokking, gradient and weight decay align, and the edge becomes a compression axis.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We decompose the spectral edge -- the dominant direction of the Gram matrix of parameter updates -- into its gradient and weight-decay components during grokking in two sequence tasks (Dyck-1 and SCAN). We find a sharp two-phase lifecycle: before grokking the edge is gradient-driven and functionally active; at grokking, gradient and weight decay align, and the edge becomes a compression axis that is perturbation-flat yet ablation-critical (>4000x more impactful than random directions). Three universality classes emerge (functional, mixed, compression), predicted by the gap flow equation. Nonlinear probes show information is re-encoded, not lost (MLP $R^2=0.99$ where linear $R^2=0.86$), and removing weight decay post-grok reverses compression while preserving the algorithm.

Related papers

• DeepLoop: Depth Scaling for Looped Transformers [91.7922038545625]
  Looped Transformers scale sequential computation by applying a compact stack of physical blocks for multiple rounds.<n>We formalize this tied-depth effect through a first-order perturbation bound controlled by a visit-alignment coefficient.<n>The resulting method, textbfDeepLoop, keeps the Post-LN DeepNorm architecture and sets $= (2N)1/2$ and $=(8N)-1/2$ for unrolled depth.
  arXiv  Detail & Related papers  (2026-07-15T06:37:05Z)
• Conservation Law Breaking at the Edge of Stability: A Spectral Theory of Non-Convex Neural Network Optimization [0.0]
  We show that gradient descent reliably finds good solutions in non-dimensional network optimization.<n>We identify two regimes separated by a width-dependent dynamical transition.<n>All predictions are validated across 23 experiments.
  arXiv  Detail & Related papers  (2026-04-08T10:41:24Z)
• The Spectral Lifecycle of Transformer Training: Transient Compression Waves, Persistent Spectral Gradients, and the Q/K--V Asymmetry [4.28787537081191]
  We present the first systematic study of weight matrix singular value spectra emphduring transformer pretraining.<n>We discover three phenomena: textbf(1)Transient Compression Waves: stable rank compression propagates as a traveling wave from early to late layers.<n>textbf(2)Persistent Spectral Gradients: the power-law exponent$$ develops a permanent depth gradient forming a non-monotonic inverted-U in deeper models.
  arXiv  Detail & Related papers  (2026-04-03T08:58:53Z)
• Robust Automatic Differentiation of Square-Root Kalman Filters via Gramian Differentials [3.542663650865005]
  Square-root Kalman filters propagate state covariances in Cholesky-factor form for numerical stability.<n>Their core operation, triangularization of a matrix $M in mathbbRn times m$, is computed via a QR decomposition in practice.<n>We extend it to rank-deficient inputs via a two-component decomposition: a column-space term based on the Moore--Penrose pseudoinverse, and a null-space correction for perturbations outside the column space of $M$.
  arXiv  Detail & Related papers  (2026-03-13T19:55:08Z)
• Marginals Before Conditionals [0.15229257192293197]
  We construct a minimal task that isolates conditional learning in neural networks.<n>The model learns the marginal P(A | B) first, producing a plateau at exactly log K, before acquiring the full conditional.
  arXiv  Detail & Related papers  (2026-03-10T05:51:01Z)
• Robust Layerwise Scaling Rules by Proper Weight Decay Tuning [50.11170157029911]
  In modern scale-invariant architectures, training quickly enters an degrading-governed steady state.<n>We introduce a weight-decay scaling rule for AdamW that preserves sublayer gain across widths.<n>Our results extend $mu$P beyond the near-init regime by explicitly controlling the steady-state scales set by parameters.
  arXiv  Detail & Related papers  (2025-10-17T02:58:35Z)
• Non-stationary Online Learning for Curved Losses: Improved Dynamic Regret via Mixability [65.99855403424979]
  We show that dynamic regret can be substantially improved by leveraging the concept of mixability.<n>We demonstrate that an exponential-weight method with fixed-share updates achieves an $mathcalO(d T2/3 P_T2/3 log T)$ dynamic regret for mixable losses.
  arXiv  Detail & Related papers  (2025-06-12T12:00:08Z)
• On the Saturation Effects of Spectral Algorithms in Large Dimensions [14.63552944833659]
  Main focus of this paper is to explore the saturation effects for a large class of spectral algorithms in large dimensional settings.<n>We show that the saturation effect occurs in large dimensional setting as long as the source condition $s>tau$ while it occurs in fixed dimensional setting as long as $s>2tau$
  arXiv  Detail & Related papers  (2025-03-01T14:21:38Z)
• Extended Linear Regression: A Kalman Filter Approach for Minimizing Loss via Area Under the Curve [0.0]
  This research enhances linear regression models by integrating a Kalman filter and analysing curve areas to minimize loss. The goal is to develop an optimal linear regression equation using gradient descent (SGD) for weight updating.
  arXiv  Detail & Related papers  (2023-08-23T17:50:57Z)
• Computationally Efficient Horizon-Free Reinforcement Learning for Linear Mixture MDPs [111.75736569611159]
  We propose the first computationally efficient horizon-free algorithm for linear mixture MDPs. Our algorithm adapts a weighted least square estimator for the unknown transitional dynamic. This also improves upon the best-known algorithms in this setting when $sigma_k2$'s are known.
  arXiv  Detail & Related papers  (2022-05-23T17:59:18Z)
• Large Scale Private Learning via Low-rank Reparametrization [77.38947817228656]
  We propose a reparametrization scheme to address the challenges of applying differentially private SGD on large neural networks. We are the first able to apply differential privacy on the BERT model and achieve an average accuracy of $83.9%$ on four downstream tasks.
  arXiv  Detail & Related papers  (2021-06-17T10:14:43Z)
• Agnostic Learning of Halfspaces with Gradient Descent via Soft Margins [92.7662890047311]
  We show that gradient descent finds halfspaces with classification error $tilde O(mathsfOPT1/2) + varepsilon$ in $mathrmpoly(d,1/varepsilon)$ time and sample complexity.
  arXiv  Detail & Related papers  (2020-10-01T16:48:33Z)
• On Biased Compression for Distributed Learning [55.89300593805943]
  We show for the first time that biased compressors can lead to linear convergence rates both in the single node and distributed settings. We propose several new biased compressors with promising theoretical guarantees and practical performance.
  arXiv  Detail & Related papers  (2020-02-27T19:52:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.