Exact Attention Sensitivity and the Geometry of Transformer Stability

• URL: http://arxiv.org/abs/2602.18849v1
• Date: Sat, 21 Feb 2026 14:25:24 GMT
• Title: Exact Attention Sensitivity and the Geometry of Transformer Stability
• Abstract summary: Despite powering modern AI, transformers remain mysteriously brittle to train.<n>We develop a stability theory that explains why pre-LayerNorm works, why DeepNorm uses $N-1/4$ scaling, and why warmup is necessary.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Despite powering modern AI, transformers remain mysteriously brittle to train. We develop a stability theory that explains why pre-LayerNorm works, why DeepNorm uses $N^{-1/4}$ scaling, and why warmup is necessary, all from first principles. Our framework has two pillars: (1) We derive the \emph{exact} operator norm of the softmax Jacobian, $\|J_{softmax}(u/τ)\|_{\infty\to 1} = θ(p)/τ$, where the balanced-mass factor $θ(p)\in[0,1]$ quantifies attention sensitivity. (2) We introduce a block-$\infty$/RMS geometry aligned with tokenwise computation, yielding Lipschitz bounds independent of sequence length. Using this framework, we prove that pre-LN preserves identity gradient paths while post-LN compounds LayerNorm Jacobians exponentially with depth, and we show that DeepNorm's $N^{-1/4}$ emerges from the quartic structure of attention's four projection matrices. We validate our theory on 774M-parameter models and find that, contrary to the intuition that attention sharpens during training to reduce sensitivity, $θ(p) \approx 1$ persists throughout. Transformer stability arises entirely from architectural gradient flow, not from attention dynamics. This finding changes how we reason about training: the architecture itself must handle sensitivity, not learned attention patterns.

Related papers

• Sharp Stability Threshold and Certification for Designing Stable Residual Architectures [4.781989580907985]
  We propose a sharp stability threshold on the input-magnitude exponent of every residual block's velocity field.<n>The threshold $q = 1$ is established via two independent arguments.<n>An arithmetic of input-magnitude exponents under the five operations that build residual blocks enables efficient certification of $q_k le 1$ at the level of architectural primitives.
  arXiv  Detail & Related papers  (2026-07-16T05:13:44Z)
• Wavelet Variance Equipartition as a Threshold for World-Model Quality and Quantum Kernel TN-Simulability [0.0]
  We identify the wavelet scaling exponent $$ as a critical diagnostic.<n>We establish $= 1/2$ as a sharp transition boundary for the classical simulability of amplitude-encoded quantum kernels.<n>We prove this variance scales strictly as $Var[X] = (d-2)$.
  arXiv  Detail & Related papers  (2026-05-12T05:41:12Z)
• Architecture Shape Governs QNN Trainability: Jacobian Null Space Growth and Parameter Efficiency [3.1094222618006664]
  We show that trainability varies substantially with architecture shape $(N,L)$ at fixed $E$.<n>We identify structural rank deficiency of the coefficient matching Jacobian $J$ as the mechanism responsible.
  arXiv  Detail & Related papers  (2026-05-07T09:52:45Z)
• Closing the Theory-Practice Gap in Spiking Transformers via Effective Dimension [13.891522069967507]
  Spiking transformers achieve competitive accuracy with conventional transformers while offering $38$-$57times$ energy efficiency on neuromorphic hardware.<n>This paper establishes the first comprehensive expressivity theory for spiking self-attention.
  arXiv  Detail & Related papers  (2026-04-17T07:15:53Z)
• Rank-Aware Spectral Bounds on Attention Logits for Stable Low-Precision Training [0.0]
  Attention scores in transformers are bilinear forms $S_ij = x_itop M x_j / sqrtd_h$ whose maximum magnitude governs overflow risk in low-precision training.<n>We derive a emphrank-aware concentration inequality: when the interaction matrix $M = WQ WKtop$ has rank $r ll d$, tail probabilities for $max_i,j|S_ij|$ decay as $exp(-d22/(
  arXiv  Detail & Related papers  (2026-02-21T14:29:22Z)
• Momentum Attention: The Physics of In-Context Learning and Spectral Forensics for Mechanistic Interpretability [0.0]
  We introduce Momentum Attention, a symplectic augmentation embedding physical priors via the kinematic difference operator.<n>We formalize an Orthogonality Theorem proving that DC (semantic) and AC (mechanistic) signals segregate into frequency bands when Low-Pass RoPE interacts with High-Pass Momentum.
  arXiv  Detail & Related papers  (2026-02-03T21:31:36Z)
• The Spectral Dimension of NTKs is Constant: A Theory of Implicit Regularization, Finite-Width Stability, and Scalable Estimation [0.0]
  We prove a constant-limit law $lim_ntoinfty mathbbE[r_texteff(K_n)] = mathbbE[k(x, x)]2.<n>We further establish finite-width stability: if the finite-width NTK deviates in operator norm by $O_p(m-1/2)$, then $r_texteff$ changes by $O_p(m-1/2)$.<n>We design a scalable estimator using random
  arXiv  Detail & Related papers  (2025-11-30T12:14:21Z)
• INC: An Indirect Neural Corrector for Auto-Regressive Hybrid PDE Solvers [61.84396402100827]
  We propose the Indirect Neural Corrector ($mathrmINC$), which integrates learned corrections into the governing equations.<n>$mathrmINC$ reduces the error amplification on the order of $t-1 + L$, where $t$ is the timestep and $L$ the Lipschitz constant.<n>We test $mathrmINC$ in extensive benchmarks, covering numerous differentiable solvers, neural backbones, and test cases ranging from a 1D chaotic system to 3D turbulence.
  arXiv  Detail & Related papers  (2025-11-16T20:14:28Z)
• Numerical Fragility in Transformers: A Layer-wise Theory for Explaining, Forecasting, and Mitigating Instability [0.0]
  We give a first-order, module-wise theory that predicts when and where errors grow.<n>For self-attention we derive a per-layer bound that factorizes into three interpretable diagnostics.<n>We also introduce a precision- and width-aware LayerNorm indicator $rho_rm LN$ with a matching first-order bound.
  arXiv  Detail & Related papers  (2025-10-17T01:03:02Z)
• Convergence Rate Analysis of LION [54.28350823319057]
  LION converges iterations of $cal(sqrtdK-)$ measured by gradient Karush-Kuhn-T (sqrtdK-)$. We show that LION can achieve lower loss and higher performance compared to standard SGD.
  arXiv  Detail & Related papers  (2024-11-12T11:30:53Z)
• Neural network learns low-dimensional polynomials with SGD near the information-theoretic limit [75.4661041626338]
  We study the problem of gradient descent learning of a single-index target function $f_*(boldsymbolx) = textstylesigma_*left(langleboldsymbolx,boldsymbolthetarangleright)$<n>We prove that a two-layer neural network optimized by an SGD-based algorithm learns $f_*$ with a complexity that is not governed by information exponents.
  arXiv  Detail & Related papers  (2024-06-03T17:56:58Z)
• Transformers as Support Vector Machines [54.642793677472724]
  We establish a formal equivalence between the optimization geometry of self-attention and a hard-margin SVM problem. We characterize the implicit bias of 1-layer transformers optimized with gradient descent. We believe these findings inspire the interpretation of transformers as a hierarchy of SVMs that separates and selects optimal tokens.
  arXiv  Detail & Related papers  (2023-08-31T17:57:50Z)
• Precision of quantum simulation of all-to-all coupling in a local architecture [0.0]
  We find an analytic relation between the values $J_ij$ of the desired interaction and the parameters of the 2d circuit. For the relative error to be a constant $epsilon$, one requires an energy scale growing as $n6$ in the number of qubits. Our proof is based on the Schrieffer-Wolff transformation and generalizes to any hardware.
  arXiv  Detail & Related papers  (2023-02-05T18:54:28Z)
• Optimal Query Complexities for Dynamic Trace Estimation [59.032228008383484]
  We consider the problem of minimizing the number of matrix-vector queries needed for accurate trace estimation in the dynamic setting where our underlying matrix is changing slowly. We provide a novel binary tree summation procedure that simultaneously estimates all $m$ traces up to $epsilon$ error with $delta$ failure probability. Our lower bounds (1) give the first tight bounds for Hutchinson's estimator in the matrix-vector product model with Frobenius norm error even in the static setting, and (2) are the first unconditional lower bounds for dynamic trace estimation.
  arXiv  Detail & Related papers  (2022-09-30T04:15:44Z)
• Bounding the Width of Neural Networks via Coupled Initialization -- A Worst Case Analysis [121.9821494461427]
  We show how to significantly reduce the number of neurons required for two-layer ReLU networks. We also prove new lower bounds that improve upon prior work, and that under certain assumptions, are best possible.
  arXiv  Detail & Related papers  (2022-06-26T06:51:31Z)
• Generalized moduli of continuity under irregular or random deformations via multiscale analysis [0.0]
  We prove that for signals in multiresolution approximation spaces $U_s$ at scale $s$, stability in $L2$ holds in the regime $|tau|_Linfty/sll 1$.<n>Instability occurs when $|tau|_Linfty/sgg 1$, and we provide a sharp upper bound for the growth rate.
  arXiv  Detail & Related papers  (2021-04-24T16:16:30Z)
• Entanglement scaling for $\lambda\phi_2^4$ [0.0]
  We show that the order parameter $phi$, the correlation length $xi$ and quantities like $phi3$ and the entanglement entropy exhibit useful double scaling properties. We find the value $alpha_c=11.09698(31)$ for the critical point, improving on previous results.
  arXiv  Detail & Related papers  (2021-04-21T14:43:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.