Spectral Edge Dynamics of Training Trajectories: Signal--Noise Geometry Across Scales

• URL: http://arxiv.org/abs/2603.15678v2
• Date: Wed, 18 Mar 2026 22:22:36 GMT
• Title: Spectral Edge Dynamics of Training Trajectories: Signal--Noise Geometry Across Scales
• Abstract summary: We show that transformer training trajectories evolve within only a few coherent directions.<n>In companion work, the same spectral geometry provides early-warning signals of grokking.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Despite hundreds of millions of parameters, transformer training trajectories evolve within only a few coherent directions. We introduce Spectral Edge Dynamics (SED) to quantify this structure: a rolling-window SVD of parameter updates reveals a sharp boundary -- the spectral edge -- between coherent optimization directions and stochastic noise, identified via the maximum consecutive singular value ratio $σ_k / σ_{k+1}$. Across a 51M-parameter TinyStories model (4 seeds) and GPT-2 124M under distribution shift, the spectral edge exhibits a universal three-phase pattern (rise, plateau, collapse). The effective signal rank adapts to task complexity ($k^* = 2$ at 51M, $k^* = 3$ at 124M), and the directional coupling between spectral geometry and validation loss reverses with window size -- a lag flip reflecting the timescale of trajectory integration. Johnson--Lindenstrauss projection to $d = 10W$ dimensions (e.g., $d = 100$ for $W = 10$) preserves the spectral gap within $5.7\%$, making the framework applicable to models of arbitrary scale. In companion work, the same spectral geometry provides early-warning signals of grokking -- predicting generalization $600$--$1{,}700$ steps before it occurs across modular arithmetic, Dyck languages, and the SCAN benchmark.

Related papers

• Two-Stage Deformable-Convolutional Inverse Design of Nanophotonic Absorbers from Optical Spectra [2.4149105714758545]
  Data-driven inverse design enables efficient generation of nanophotonic structures with prescribed optical responses.<n>This work presents a two-stage deformable-convolutional framework for reconstructing metal--insulator--metal resonator from 80-dimensional absorption spectra.
  arXiv  Detail & Related papers  (2026-08-12T09:47:06Z)
• Round-Trip Consistency: Bidirectional Diffusion Models Can Predict Their Own Rollout Errors [51.56484100374058]
  Autoregressive models accumulate error over long rollouts, yet at deployment there is no ground truth to measure it against.<n>We train a single conditional latent diffusion model that steps a dynamical system forward or backward in time via a direction flag.<n>We show that this bidirectionality supplies a measurement-free test-time error signal.
  arXiv  Detail & Related papers  (2026-08-01T13:49:46Z)
• Dead-Direction Signatures: A Cheap Spectral Reading of Singular Complexity [5.67642958049511]
  Singular learning theory characterises the complexity of a deep network through the geometry of its loss singularities.<n>We introduce DeadDirection Signatures (DDS), a family of cheap closed-form spectral readings of singular structure.<n>Each reads a network's activation matrix or per-sample-gradient FisherGram at a chosen layer, replacing the posterior chain spectral linear algebra.
  arXiv  Detail & Related papers  (2026-06-19T06:49:09Z)
• Closed-Form Spectral Regularization for Multi-Task Model Merging [96.82449201305234]
  Model merging combines several independently fine-tuned experts into a single multi-task model without any training data.<n>State-of-the-art merging methods formulate merging as a layer-wise interference problem.<n>We propose SWUDI, a closed-form method that combines a soft exponential filter, which matches the gradient-flow trajectory of iterative descent.
  arXiv  Detail & Related papers  (2026-06-05T14:00:47Z)
• Concepts Whisper While Syntax Shouts: Spectral Anti-Concentration and the Dual Geometry of Transformer Representations [0.0]
  We find anti-concentration is observed in residual-stream difference-of-means vectors across five architecture families.<n>Results suggest transformers may rotate semantic content into spectrally quiet regions during contextualized processing.
  arXiv  Detail & Related papers  (2026-05-02T21:20:55Z)
• Optimal Decay Spectra for Linear Recurrences [6.508284558109273]
  PoST integrates into any diagonal linear recurrence without overhead.<n>We instantiate it across Mamba RWKV-7, Gated DeltaNet, Gated Linear Attention, and RetNet.
  arXiv  Detail & Related papers  (2026-04-08T23:54:57Z)
• The Spectral Edge Thesis: A Mathematical Framework for Intra-Signal Phase Transitions in Neural Network Training [0.0]
  We show that phase transitions in neural network training are controlled by the spectral gap of the rolling-window Gram matrix of parameter updates.<n>Adiabatic parameter $mathcalA = |G|_F / (, g2)$ controls circuit stability: $mathcalA ll 1$ (plateau), $mathcalA sim 1$ (phase transition), $mathcalA gg 1$ (forgetting)
  arXiv  Detail & Related papers  (2026-03-30T20:10:22Z)
• GST-VLA: Structured Gaussian Spatial Tokens for 3D Depth-Aware Vision-Language-Action Models [6.2676602262188625]
  VLA models encode visual observations as 2D patch tokens with no intrinsic geometric structure.<n>We introduce GST-VLA with two contributions.<n>First, the Gaussian Spatial Tokenizer ( GST) converts frozen dense depth and frozen semantic patch features into 3D primitives.<n>Second, 3D Depth-Aware Chain-of-Thought (DA-CoT) reasoning supervises four structured intermediate spatial thoughts.
  arXiv  Detail & Related papers  (2026-03-10T01:39:38Z)
• Do Foundation Models Know Geometry? Probing Frozen Features for Continuous Physical Measurement [0.0]
  Vision-language models encode geometry that their text pathway fails to express.<n>LoRA fine-tuning (r=16, 2,000 images) narrows this gap to 6.5 degrees.<n>These findings enable a single frozen backbone to function as a multi-task geometric sensor.
  arXiv  Detail & Related papers  (2026-03-06T16:48:27Z)
• Anatomy of Capability Emergence: Scale-Invariant Representation Collapse and Top-Down Reorganization in Neural Networks [1.5567685129899713]
  We track five geometric measures across five model scales.<n>We find that training begins with a universal representation collapse to task-specific floors.<n>On Pythia, global geometric patterns replicate but per-task precursor signals do not.
  arXiv  Detail & Related papers  (2026-02-17T20:39:02Z)
• Structure-Informed Estimation for Pilot-Limited MIMO Channels via Tensor Decomposition [51.56484100374058]
  This paper formulates pilot-limited channel estimation as low-rank tensor completion from sparse observations.<n>Experiments on synthetic channels demonstrate 10-20,dB normalized mean-square error (NMSE) improvement over least-squares (LS)<n> evaluations on DeepMIMO ray-tracing channels show 24-44% additional NMSE reduction over pure tensor-based methods.
  arXiv  Detail & Related papers  (2026-02-03T23:38:05Z)
• Robust inverse material design with physical guarantees using the Voigt-Reuss Net [0.0]
  We propose a spectrally normalized surrogate for forward and inverse mechanical homogenization with hard physical guarantees.<n>In 3D linear elasticity on an open dataset of biphasic microstructures, a fully connected Voigt-Reuss net trained on $>!7.5times 105$ FFT-based labels with 236 isotropy-in descriptors.<n>Overall, the Voigt-Reuss net unifies accurate, physically admissible forward prediction with large-batch, constraint-consistent inverse design.
  arXiv  Detail & Related papers  (2025-11-14T15:17:37Z)
• Robust Layerwise Scaling Rules by Proper Weight Decay Tuning [50.11170157029911]
  In modern scale-invariant architectures, training quickly enters an degrading-governed steady state.<n>We introduce a weight-decay scaling rule for AdamW that preserves sublayer gain across widths.<n>Our results extend $mu$P beyond the near-init regime by explicitly controlling the steady-state scales set by parameters.
  arXiv  Detail & Related papers  (2025-10-17T02:58:35Z)
• Near-Optimal Online Learning for Multi-Agent Submodular Coordination: Tight Approximation and Communication Efficiency [52.60557300927007]
  We present a $textbfMA-OSMA$ algorithm to transfer the discrete submodular problem into a continuous optimization.<n>We also introduce a projection-free $textbfMA-OSEA$ algorithm, which effectively utilizes the KL divergence by mixing a uniform distribution.<n>Our algorithms significantly improve the $(frac11+c)$-approximation provided by the state-of-the-art OSG algorithm.
  arXiv  Detail & Related papers  (2025-02-07T15:57:56Z)
• Transformers as Support Vector Machines [54.642793677472724]
  We establish a formal equivalence between the optimization geometry of self-attention and a hard-margin SVM problem. We characterize the implicit bias of 1-layer transformers optimized with gradient descent. We believe these findings inspire the interpretation of transformers as a hierarchy of SVMs that separates and selects optimal tokens.
  arXiv  Detail & Related papers  (2023-08-31T17:57:50Z)
• Spectral density estimation with the Gaussian Integral Transform [91.3755431537592]
  spectral density operator $hatrho(omega)=delta(omega-hatH)$ plays a central role in linear response theory. We describe a near optimal quantum algorithm providing an approximation to the spectral density.
  arXiv  Detail & Related papers  (2020-04-10T03:14:38Z)
• Minimum optical depth multiport interferometers for approximating arbitrary unitary operations and pure states [37.69303106863453]
  We address the problem of, using multiport interferometers, approximating with a given infidelity any pure state preparation and any unitary operation.<n>By means of numerical calculations, we show that pure states, in any dimension $d$, can be prepared with infidelity.
  arXiv  Detail & Related papers  (2020-02-04T15:40:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.