Spectral Outliers Reveal Dominant Learned Structure in Transformer Attention

• URL: http://arxiv.org/abs/2608.07921v1
• Date: Sat, 08 Aug 2026 04:56:11 GMT
• Title: Spectral Outliers Reveal Dominant Learned Structure in Transformer Attention
• Abstract summary: We apply Marchenko-Pastur (MP) random matrix theory to pre-trained attention weights in order to separate each projection matrix into a random-like bulk and a set of spectral outliers.<n>We validate this decomposition causally: zeroing the MP-identified outliers (signal) in Mistral-7B drives HellaSwag, MMLU, and PIQA close to random-chance performance.
• Score: 0.8971132850029493
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We apply Marchenko-Pastur (MP) random matrix theory to pre-trained attention weights in order to separate each projection matrix into a random-like bulk and a set of spectral outliers. We validate this decomposition causally: zeroing the MP-identified outliers (signal) in Mistral-7B drives HellaSwag, MMLU, and PIQA close to random-chance performance, whereas zeroing a count-matched subset of bulk singular values causes smaller but non-negligible degradation. Across 11 pre-trained transformers we identify five recurring patterns: spectral outliers encode a dominant component of the learned structure; Q projections carry the most outliers; V projections under grouped-query attention lack a clean signal/noise separation; entry-level outliers form structured row-bands in Q and column-bands in O; and specific residual-stream dimensions persist as band outliers across layers in K and O. We close by outlining how these observations could inform parameter-efficient fine-tuning and structured pruning.

Related papers

• Multi-kernel spectral clustering: Entrywise eigenvector perturbation bounds and exact recovery [5.506184984859199]
  Kernel spectral clustering with a single bandwidth can be inadequate for data exhibiting multiple characteristic pairwise-distance scales.<n>We address this issue through a multi- kernel formulation that aggregates kernels with different bandwidths.<n>We develop a rigorous theoretical analysis of the resulting method under a general high-dimensional, multi-scale mixture model.
  arXiv  Detail & Related papers  (2026-08-09T13:34:27Z)
• RamanPFN: learning from Raman spectral structure with a tabular foundation model [5.156304764307903]
  Raman spectroscopy enables non-destructive, label-free molecular characterization across materials science, biomedicine and process monitoring.<n> Predictive Raman datasets often contain few labelled spectra and thousands of ordered wavenumbers.<n>We present RamanPFN, a spectral representation framework that encodes dependencies before TabPFN inference.
  arXiv  Detail & Related papers  (2026-08-03T12:37:55Z)
• Relevant and Irrelevant: A Renormalization Group Analysis of Transformer Attention [16.36212521143563]
  We treat the Transformer's attention mechanism as a perturbation of the trained residual-stack fixed point.<n>Testable predictions are made for the fixed-point geometry, effective rank profile, layer specificity, and perturbation decay spectrum.
  arXiv  Detail & Related papers  (2026-07-16T20:42:02Z)
• Many Circuits, One Mechanism: Input Variation and Evaluation Granularity in Circuit Discovery [49.50794188352666]
  We show that structural differences between circuits are not sufficient evidence for distinct mechanisms.<n>Standard evaluation practice obscures this pattern, while edge-level evaluation reveals the many-to-one mapping from structure to function.
  arXiv  Detail & Related papers  (2026-06-04T15:10:14Z)
• Coverage Analysis of Rydberg Atom Quantum Receiver Arrays: A Stochastic Geometry Approach [55.47164192369501]
  Rydberg atomic quantum receivers (RAQRs) offer quantum-limited sensitivity and broadband tunability.<n>In dense deployments, aggregate interference can push atomic transducer out of its small-signal regime.<n>This paper addresses the question by embedding the RAQR front end into a geometry coverage analysis.
  arXiv  Detail & Related papers  (2026-05-22T04:05:13Z)
• On the Spectral Flattening of Quantized Embeddings [25.64641307046705]
  Training Large Language Models at ultra-low precision is critically impeded by instability rooted in the conflict between discrete quantization constraints and the intrinsic heavy-tailed spectral nature of linguistic data.<n>This work not only quantifies the spectral sensitivity of LLMs but also establishes spectral fidelity as a necessary condition for stable low-bit optimization.
  arXiv  Detail & Related papers  (2026-02-01T02:21:53Z)
• The Homogeneity Trap: Spectral Collapse in Doubly-Stochastic Deep Networks [1.7523718031184992]
  We identify a critical spectral degradation phenomenon inherent to structure-preserving deep architectures.<n>We show that maximum-entropy bias drives the mixing operator towards the uniform barycenter, suppressing the subdominant singular value .<n>We derive a spectral bound linking to the network's effective depth, showing that high-entropy constraints restrict feature transformation to a shallow receptive field.
  arXiv  Detail & Related papers  (2026-01-05T13:09:42Z)
• Unsupervised Conformal Inference: Bootstrapping and Alignment to Control LLM Uncertainty [49.19257648205146]
  We propose an unsupervised conformal inference framework for generation.<n>Our gates achieve close-to-nominal coverage and provide tighter, more stable thresholds than split UCP.<n>The result is a label-free, API-compatible gate for test-time filtering.
  arXiv  Detail & Related papers  (2025-09-26T23:40:47Z)
• Avoided-crossings, degeneracies and Berry phases in the spectrum of quantum noise through analytic Bloch-Messiah decomposition [49.1574468325115]
  "analytic Bloch-Messiah decomposition" provides approach for characterizing dynamics of quantum optical systems.<n>We show that avoided crossings arise naturally when a single parameter is varied, leading to hypersensitivity of the singular vectors.<n>We highlight the possibility of programming the spectral response of photonic systems through the deliberate design of avoided crossings.
  arXiv  Detail & Related papers  (2025-04-29T13:14:15Z)
• Three-fold way of entanglement dynamics in monitored quantum circuits [68.8204255655161]
  We investigate the measurement-induced entanglement transition in quantum circuits built upon Dyson's three circular ensembles. We obtain insights into the interplay between the local entanglement generation by the gates and the entanglement reduction by the measurements.
  arXiv  Detail & Related papers  (2022-01-28T17:21:15Z)
• Sparse Quantized Spectral Clustering [85.77233010209368]
  We exploit tools from random matrix theory to make precise statements about how the eigenspectrum of a matrix changes under such nonlinear transformations. We show that very little change occurs in the informative eigenstructure even under drastic sparsification/quantization.
  arXiv  Detail & Related papers  (2020-10-03T15:58:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.