Momentum Attention: The Physics of In-Context Learning and Spectral Forensics for Mechanistic Interpretability

• URL: http://arxiv.org/abs/2602.04902v2
• Date: Sat, 07 Feb 2026 21:56:02 GMT
• Title: Momentum Attention: The Physics of In-Context Learning and Spectral Forensics for Mechanistic Interpretability
• Abstract summary: We introduce Momentum Attention, a symplectic augmentation embedding physical priors via the kinematic difference operator.<n>We formalize an Orthogonality Theorem proving that DC (semantic) and AC (mechanistic) signals segregate into frequency bands when Low-Pass RoPE interacts with High-Pass Momentum.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The Mechanistic Interpretability (MI) program has mapped the Transformer as a precise computational graph. We extend this graph with a conservation law and time-varying AC dynamics, viewing it as a physical circuit. We introduce Momentum Attention, a symplectic augmentation embedding physical priors via the kinematic difference operator $p_t = q_t - q_{t-1}$, implementing the symplectic shear $\hat{q}_t = q_t + γp_t$ on queries and keys. We identify a fundamental Symplectic-Filter Duality: the physical shear is mathematically equivalent to a High-Pass Filter. This duality is our cornerstone contribution -- by injecting kinematic momentum, we sidestep the topological depth constraint ($L \geq 2$) for induction head formation. While standard architectures require two layers for induction from static positions, our extension grants direct access to velocity, enabling Single-Layer Induction and Spectral Forensics via Bode Plots. We formalize an Orthogonality Theorem proving that DC (semantic) and AC (mechanistic) signals segregate into orthogonal frequency bands when Low-Pass RoPE interacts with High-Pass Momentum. Validated through 5,100+ controlled experiments (documented in Supplementary Appendices A--R and 27 Jupyter notebooks), our 125M Momentum model exceeds expectations on induction-heavy tasks while tracking a 350M baseline within $\sim$2.9% validation loss. Dedicated associative recall experiments reveal a scaling law $γ^* = 4.17 \times N^{-0.74}$ establishing momentum-depth fungibility. We offer this framework as a complementary analytical toolkit connecting Generative AI, Hamiltonian Physics, and Signal Processing.

Related papers

• The HALO Engine: $\mathcal{O}(1)$-Step Compilation and Localized String Rupture for Lattice Gauge Theories on Quantum Hardware [0.0]
  We introduce the Hardware-Aware Lattice Optimization (HALO) compiler, which executes global time-evolution steps in an immutable $mathcalO(1)$ circuit depth per Trotter step.<n>We execute the non-perturbative dynamics of a heavily stretched $L=15$ meson string on a 16-qubit superconducting transmon processor.<n>We map the dynamical phase diagram of the mesoscopic lattice, pinpointing the effective confinement phase boundary at precisely $g_c = 1.0$.
  arXiv  Detail & Related papers  (2026-08-14T13:07:47Z)
• GQA-μP: The maximal parameterization update for grouped query attention [64.99972873396649]
  We show how to derivation P scalings for grouped-query attention (GQA)<n>We also demonstrate the efficacy of our theoretical derivations by showing learning rate transfer across the GQA repetition hyperparameter.
  arXiv  Detail & Related papers  (2026-05-14T18:03:16Z)
• Closing the Theory-Practice Gap in Spiking Transformers via Effective Dimension [13.891522069967507]
  Spiking transformers achieve competitive accuracy with conventional transformers while offering $38$-$57times$ energy efficiency on neuromorphic hardware.<n>This paper establishes the first comprehensive expressivity theory for spiking self-attention.
  arXiv  Detail & Related papers  (2026-04-17T07:15:53Z)
• Topological Characterization of Churn Flow and Unsupervised Correction to the Wu Flow-Regime Map in Small-Diameter Vertical Pipes [0.0]
  We introduce the first topology-based characterization using Characteristic Surfaces (ECS)<n>This work provides the first mathematical definition of churn flow and demonstrates that unsupervised topological descriptors can challenge and correct mechanistic models.
  arXiv  Detail & Related papers  (2026-04-07T17:59:15Z)
• Unsupervised Discovery of Intermediate Phase Order in the Frustrated $J_1$-$J_2$ Heisenberg Model via Prometheus Framework [0.0]
  We apply the Prometheus variational autoencoder framework to explore the $J_1$-$J$ phase diagram.<n>We identify the structure factor $S(,)$ and $S(,)$ as the dominant order parameters.<n>This work establishes a scalable pathway for applying machine learning to frustrated quantum systems.
  arXiv  Detail & Related papers  (2026-02-25T00:44:51Z)
• Exact Attention Sensitivity and the Geometry of Transformer Stability [0.0]
  Despite powering modern AI, transformers remain mysteriously brittle to train.<n>We develop a stability theory that explains why pre-LayerNorm works, why DeepNorm uses $N-1/4$ scaling, and why warmup is necessary.
  arXiv  Detail & Related papers  (2026-02-21T14:25:24Z)
• Gradient Dynamics of Attention: How Cross-Entropy Sculpts Bayesian Manifolds [0.4779196219827507]
  We show how cross-entropy training reshapes attention scores and value vectors in a transformer attention head.<n>Our core result is an emphadvantage-based routing law for attention scores.<n>We show that this coupled specialization behaves like a two-timescale EM procedure.
  arXiv  Detail & Related papers  (2025-12-27T05:31:44Z)
• Analytical blueprint for 99.999% fidelity X-gates on present superconducting hardware under strong driving [4.406638884109584]
  We numerically demonstrate gate infidelities below $10-5$ for a 7ns $$-rotation when incorporating existing decoherence rates.<n>We also answer long-standing questions about the optimal values of the DRAG prefactor as well as constant detuning.
  arXiv  Detail & Related papers  (2025-12-22T22:47:43Z)
• Renormalizable Spectral-Shell Dynamics as the Origin of Neural Scaling Laws [2.779943773196378]
  We show that deep-network training obeys a simple macroscopic structure despite highly nonlinear optimization dynamics.<n>For mean-squared error loss, the training error evolves as $dot e_t=-M(t)e_t$ with $M(t)=J_(t)J_(t)!*$, a time-dependent self-adjoint operator induced by the network Jacobian.<n>This framework explains neural scaling laws and double descent, and unifies lazy (NTK-like) training and feature learning as two limits of the same spectral-shell
  arXiv  Detail & Related papers  (2025-12-11T08:38:46Z)
• Mimicking the Physicist's Eye:A VLM-centric Approach for Physics Formula Discovery [98.58830663687911]
  VIPERR-aq1 is a multimodal model that performs Visual Induction for Equation Reasoning.<n>It integrates visual perception, trajectory data, and symbolic reasoning to emulate the scientific discovery process.<n>It consistently outperforms state-of-the-art VLM baselines in accuracy and interpretability.
  arXiv  Detail & Related papers  (2025-08-24T14:34:21Z)
• Emergence and scaling laws in SGD learning of shallow neural networks [64.48316762675141]
  We study the complexity of online gradient descent (SGD) for learning a two-layer neural network with $P$ neurons on isotropic Gaussian data.<n>We provide a precise analysis of SGD dynamics for the training of a student two-layer network to minimize the mean squared error (MSE) objective.
  arXiv  Detail & Related papers  (2025-04-28T16:58:55Z)
• Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformers [54.20763128054692]
  We study how a two-attention-layer transformer is trained to perform ICL on $n$-gram Markov chain data. We prove that the gradient flow with respect to a cross-entropy ICL loss converges to a limiting model.
  arXiv  Detail & Related papers  (2024-09-09T18:10:26Z)
• Hamiltonian Mechanics of Feature Learning: Bottleneck Structure in Leaky ResNets [58.460298576330835]
  We study Leaky ResNets, which interpolate between ResNets and Fully-Connected nets depending on an 'effective depth'<n>We leverage this intuition to explain the emergence of a bottleneck structure, as observed in previous work.
  arXiv  Detail & Related papers  (2024-05-27T18:15:05Z)
• Accelerating superconductor discovery through tempered deep learning of the electron-phonon spectral function [0.0]
  We train a deep learning model to predict the electron-phonon spectral function, $alpha2F(omega)$. We then incorporate domain knowledge of the site-projected phonon density states to impose inductive bias into the model's node attributes and enhance predictions. This methodological innovation decreases the MAE to 0.18, 29 K, and 28 K, respectively yielding an MAE of 2.1 K for $T_c$.
  arXiv  Detail & Related papers  (2024-01-29T22:44:28Z)
• Preparation of excited states for nuclear dynamics on a quantum computer [117.44028458220427]
  We study two different methods to prepare excited states on a quantum computer. We benchmark these techniques on emulated and real quantum devices. These findings show that quantum techniques designed to achieve good scaling on fault tolerant devices might also provide practical benefits on devices with limited connectivity and gate fidelity.
  arXiv  Detail & Related papers  (2020-09-28T17:21:25Z)
• Continuous-time quantum walks in the presence of a quadratic perturbation [55.41644538483948]
  We address the properties of continuous-time quantum walks with Hamiltonians of the form $mathcalH= L + lambda L2$. We consider cycle, complete, and star graphs because paradigmatic models with low/high connectivity and/or symmetry.
  arXiv  Detail & Related papers  (2020-05-13T14:53:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.