Spectral Edge Dynamics Reveal Functional Modes of Learning

• URL: http://arxiv.org/abs/2604.06256v1
• Date: Mon, 06 Apr 2026 22:29:00 GMT
• Title: Spectral Edge Dynamics Reveal Functional Modes of Learning
• Abstract summary: Training dynamics during grokking concentrate along a small number of dominant update directions -- the spectral edge.<n>We show that standard mechanistic interpretability tools fail to capture these directions.<n>For modular addition, all leading directions collapse to a single Fourier mode.<n>For multiplication, the same collapse appears only in the discrete-log basis, yielding a 5.9x improvement in concentration.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Training dynamics during grokking concentrate along a small number of dominant update directions -- the spectral edge -- which reliably distinguishes grokking from non-grokking regimes. We show that standard mechanistic interpretability tools (head attribution, activation probing, sparse autoencoders) fail to capture these directions: their structure is not localized in parameter or feature space. Instead, each direction induces a structured function over the input domain, revealing low-dimensional functional modes invisible to representation-level analysis. For modular addition, all leading directions collapse to a single Fourier mode. For multiplication, the same collapse appears only in the discrete-log basis, yielding a 5.9x improvement in concentration. For subtraction, the edge spans a small multi-mode family. For $x^2+y^2$, no single harmonic basis suffices, but cross-terms of additive and multiplicative features provide a 4x variance boost, consistent with the decomposition (a+b)^2 - 2ab. Multitask training amplifies this compositional structure, with the $x^2+y^2$ spectral edge inheriting the addition circuit's characteristic frequency (2.3x concentration increase). These results suggest that training discovers low-dimensional functional modes over the input domain, whose structure depends on the algebraic symmetry of the task. These results suggest that spectral edge dynamics identify low-dimensional functional subspaces governing learning, whose representation depends on the algebraic structure of the task. Simple harmonic structure emerges only when the task admits a symmetry-adapted basis; more complex tasks require richer functional descriptions.

Related papers

• Intrinsic Structure: Spectral Identifiability for Mechanistic Interpretability [51.56484100374058]
  We prove the first identifiability theorem for a mechanistic-interpretability primitive.<n>The spectrum is an identifiable, model-intrinsic fingerprint with a stated error bar, not a legible decomposition.
  arXiv  Detail & Related papers  (2026-08-10T19:42:01Z)
• How does feature learning reshape the function space? [17.342357780996174]
  We study how the function space spanned by the features of a two-layer neural network evolves during gradient descent training.<n>Our results provide a precise function-space perspective on early-stage feature learning.
  arXiv  Detail & Related papers  (2026-05-18T00:41:20Z)
• Spectral Path Regression: Directional Chebyshev Harmonics for Interpretable Tabular Learning [0.0]
  We replace tensorised oscillations with directional harmonic modes of the form $cos(mathbfmtoparccos(mathbfx)$.<n>This representation yields a discrete spectral regression model in which complexity is controlled by selecting a small number of structured frequency vectors.
  arXiv  Detail & Related papers  (2026-04-05T12:13:59Z)
• On the Spectral Geometry of Cross-Modal Representations: A Functional Map Diagnostic for Multimodal Alignment [0.0]
  We study cross-modal alignment between independently pretrained vision (DINOv2) and language (all-MiniLM-L6-v2) encoders.<n>We find that the Laplacian eigenvalue spectra of the two encoders are quantitatively similar.<n>We term this decoupling the spectral complexity--orientation gap: models converge in how much structure they capture but not in how they organize it.
  arXiv  Detail & Related papers  (2026-03-28T04:20:54Z)
• Functorial Neural Architectures from Higher Inductive Types [5.076419064097734]
  We show that neural networks systematically fail at compositional generalization.<n>This failure is architectural: compositional generalization is equivalent to functoriality of the decoder.<n>We prove that decoders assembled by structural concatenation of independently generated segments are strict monoidal functors.
  arXiv  Detail & Related papers  (2026-03-17T05:08:30Z)
• On Counts and Densities of Homogeneous Bent Functions: An Evolutionary Approach [60.00535100780336]
  This paper examines the use of Evolutionary Algorithms (EAs) to evolve homogeneous bent Boolean functions.<n>We introduce the notion of density of homogeneous bent functions, facilitating the algorithmic design that results in finding quadratic and cubic bent functions in different numbers of variables.
  arXiv  Detail & Related papers  (2025-11-16T15:33:40Z)
• Beyond Softmax: A Natural Parameterization for Categorical Random Variables [61.709831225296305]
  We introduce the $textitcatnat$ function, a function composed of a sequence of hierarchical binary splits.<n>A rich set of experiments show that the proposed function improves the learning efficiency and yields models characterized by consistently higher test performance.
  arXiv  Detail & Related papers  (2025-09-29T12:55:50Z)
• Provable Scaling Laws of Feature Emergence from Learning Dynamics of Grokking [44.614763110719274]
  We study the phenomenon of grokking, i.e., delayed generalization.<n>We propose a novel framework that captures three key stages for the grokking behavior of 2-layer nonlinear networks.<n>Our study sheds lights on roles played by hypers such as weight decay, learning rate and sample sizes in grokking.
  arXiv  Detail & Related papers  (2025-09-25T20:08:09Z)
• DiffuMatch: Category-Agnostic Spectral Diffusion Priors for Robust Non-rigid Shape Matching [53.39693288324375]
  We show that both in-network regularization and functional map training can be replaced with data-driven methods.<n>We first train a generative model of functional maps in the spectral domain using score-based generative modeling.<n>We then exploit the resulting model to promote the structural properties of ground truth functional maps on new shape collections.
  arXiv  Detail & Related papers  (2025-07-31T16:44:54Z)
• Self-Attention as a Parametric Endofunctor: A Categorical Framework for Transformer Architectures [0.0]
  We develop a category-theoretic framework focusing on the linear components of self-attention.<n>We show that the query, key, and value maps naturally define a parametric 1-morphism in the 2-category $mathbfPara(Vect)$.<n> stacking multiple self-attention layers corresponds to constructing the free monad on this endofunctor.
  arXiv  Detail & Related papers  (2025-01-06T11:14:18Z)
• Reinforcement Learning from Partial Observation: Linear Function Approximation with Provable Sample Efficiency [111.83670279016599]
  We study reinforcement learning for partially observed decision processes (POMDPs) with infinite observation and state spaces. We make the first attempt at partial observability and function approximation for a class of POMDPs with a linear structure.
  arXiv  Detail & Related papers  (2022-04-20T21:15:38Z)
• X-volution: On the unification of convolution and self-attention [52.80459687846842]
  We propose a multi-branch elementary module composed of both convolution and self-attention operation. The proposed X-volution achieves highly competitive visual understanding improvements.
  arXiv  Detail & Related papers  (2021-06-04T04:32:02Z)
• Semiparametric Nonlinear Bipartite Graph Representation Learning with Provable Guarantees [106.91654068632882]
  We consider the bipartite graph and formalize its representation learning problem as a statistical estimation problem of parameters in a semiparametric exponential family distribution. We show that the proposed objective is strongly convex in a neighborhood around the ground truth, so that a gradient descent-based method achieves linear convergence rate. Our estimator is robust to any model misspecification within the exponential family, which is validated in extensive experiments.
  arXiv  Detail & Related papers  (2020-03-02T16:40:36Z)
• Invariant Feature Coding using Tensor Product Representation [75.62232699377877]
  We prove that the group-invariant feature vector contains sufficient discriminative information when learning a linear classifier. A novel feature model that explicitly consider group action is proposed for principal component analysis and k-means clustering.
  arXiv  Detail & Related papers  (2019-06-05T07:15:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.