Fingerprint, Not Blueprint: How Positional Schemes Set the Default Spectral Algebra of Attention

• URL: http://arxiv.org/abs/2607.06621v1
• Date: Tue, 07 Jul 2026 10:17:30 GMT
• Title: Fingerprint, Not Blueprint: How Positional Schemes Set the Default Spectral Algebra of Attention
• Abstract summary: We show that the strongest previous-token heads are spectrally rotational under RoPE.<n>We also show that over public Pythia checkpoints every head originates at the random-matrix (Ginibre) null.<n>Within the settings examined, the positional scheme sets the default spectral algebra of an attention head's solution.
• Score: 1.7042264000899534
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: The pre-softmax score of an attention head is a bilinear form $score(i,j) = x_i^T M x_j$ in a learned operator $M = W_q^T W_k$. Because M is generally non-symmetric, hence non-normal, it has a complex eigenspectrum and non-orthogonal eigenvectors, the regime where non-Hermitian and random-matrix tools apply. We ask what this spectrum encodes, at three levels for previous-token and induction circuits. Statically, across seven pretrained models spanning three positional schemes, the strongest previous-token heads are spectrally rotational under RoPE and non-rotational, or content-like, where position enters outside QK (learned-absolute and ALiBi); the model-level separation is perfect at every top-k examined (exact permutation $p=0.029$), and zeroing the per-frequency RoPE phase $Im(M_t)$ eliminates induction on a pre-identified previous-token head in all three RoPE models. Dynamically, over public Pythia checkpoints every head originates at the random-matrix (Ginibre) null; the rotational signature emerges with the behavior, not before it, and the population-median suppression that yields the final profile follows circuit formation, so the profile is a consolidated fingerprint, not a precursor. Causally, and at toy scale, no spectral channel is necessary: constrained two-layer training reroutes around every ban with capability intact, albeit at a significant formation delay (four pre-registered contrasts, $q_BH <= 0.016$). The cost structure exposes each scheme's default: imposing symmetry slows learned-absolute models by a factor of 2.9, whereas a RoPE head with a fully symmetric static M still routes directionally via the phase channel, impossible under absolute positions. Within the settings examined, the positional scheme sets the default spectral algebra of an attention head's solution: a fingerprint sculpted after function, not a hard constraint upon it.

Related papers

• $\ ext{GSF-}χ$: Global Stereochemical Fields for Chiral Graph Transformers [49.85093436361195]
  We introduce GSF-$$, a graph transformer in which stereogenic units modulate all pairwise interactions.<n>A $C$ projection separates mirror-even ECD peak counts and positions from mirror-odd peak signs.<n>GSF-$$ leads every central-ECD output and improves axial Rotation and Symbol by $12.6%$ and $7.9%$ over the strongest baseline.
  arXiv  Detail & Related papers  (2026-09-11T07:41:20Z)
• Intrinsic Structure: Spectral Identifiability for Mechanistic Interpretability [51.56484100374058]
  We prove the first identifiability theorem for a mechanistic-interpretability primitive.<n>The spectrum is an identifiable, model-intrinsic fingerprint with a stated error bar, not a legible decomposition.
  arXiv  Detail & Related papers  (2026-08-10T19:42:01Z)
• Round-Trip Consistency: Bidirectional Diffusion Models Can Predict Their Own Rollout Errors [51.56484100374058]
  Autoregressive models accumulate error over long rollouts, yet at deployment there is no ground truth to measure it against.<n>We train a single conditional latent diffusion model that steps a dynamical system forward or backward in time via a direction flag.<n>We show that this bidirectionality supplies a measurement-free test-time error signal.
  arXiv  Detail & Related papers  (2026-08-01T13:49:46Z)
• Flash EQ-Linear: Accelerating Equivariant Linear Layers via Group-wise Discrete Fourier Transform [65.89510337046077]
  Equivariant networks embed geometric symmetries as structural priors through weight sharing.<n>Existing implementations unroll the structured weights into dense matrices and dispatch them to generic dense kernels.<n>This is the first time equivariant networks strictly dominate their non-equivariant counterparts along all three axes simultaneously.
  arXiv  Detail & Related papers  (2026-07-23T12:45:21Z)
• Demixing Sparse Signals from Nonlinear Observations using Generalized Non-convex Regularization [0.0]
  We consider the recovery of a pair of sparse vectors from a limited number of nonlinear observations of their superposition.<n>We propose a regularization-based framework combining a Huberized data fidelity with generalized folded-concave penalties.<n>We derive estimation error bounds of order $sqrtslog(n)/m$ that hold at emphevery localized stationary point.
  arXiv  Detail & Related papers  (2026-07-12T07:29:39Z)
• RotateAttention: RoPE-Aware Rotation and Range Rectification for INT4 Quantized Attention in Video Generation [15.82658615412336]
  We present an efficient textbfmixed-precision INT4 FlashAttention framework tailored for textbfDiT-based video generation models with 3D RoPE.<n>Experiments show that textbfRotateAttention preserves video generation quality nearly identical to full-precision baselines while achieving up to 1.68$times$ end-to-end speedup and 2.2$times$ kernel-level acceleration.
  arXiv  Detail & Related papers  (2026-07-01T02:44:10Z)
• All Routes Lead to Collapse [0.0]
  We show that attention sinks, representation collapse, and norm stratification are not transformer-specific pathologies.<n>We give a reframing identity: softmax attention is Boltzmann-weighted aggregation over Euclidean distances with constant key norms.
  arXiv  Detail & Related papers  (2026-06-21T03:59:13Z)
• Imbuing Large Language Models with Bidirectional Logic for Robust Chain Repair [44.80087038178069]
  We introduce Teleological Reasoning Infilling (TRI), a training framework that endows decoder-only transformers with a native emphgoal-conditioned bridging capability.<n>At inference, TRI operates as a surgical repair module within a dual-system loop.<n>Experiments on three benchmarks demonstrate that TRI achieves state-of-the-art performance across all tasks, while reducing per-problem token expenditure by 31.2%.
  arXiv  Detail & Related papers  (2026-06-03T15:58:48Z)
• The Spectral Edge Thesis: A Mathematical Framework for Intra-Signal Phase Transitions in Neural Network Training [0.0]
  We show that phase transitions in neural network training are controlled by the spectral gap of the rolling-window Gram matrix of parameter updates.<n>Adiabatic parameter $mathcalA = |G|_F / (, g2)$ controls circuit stability: $mathcalA ll 1$ (plateau), $mathcalA sim 1$ (phase transition), $mathcalA gg 1$ (forgetting)
  arXiv  Detail & Related papers  (2026-03-30T20:10:22Z)
• Regularized Online RLHF with Generalized Bilinear Preferences [68.44113000390544]
  We consider the problem of contextual online RLHF with general preferences.<n>We adopt the Generalized Bilinear Preference Model to capture preferences via low-rank, skew-symmetric matrices.<n>We prove that the dual gap of the greedy policy is bounded by the square of the estimation error.
  arXiv  Detail & Related papers  (2026-02-26T15:27:53Z)
• OLion: Approaching the Hadamard Ideal by Intersecting Spectral and $\ell_{\infty}$ Implicit Biases [29.60546958677364]
  nameA combines spectral control from update directions with coordinate control from sign updates.<n>We prove convergence under a mild, empirically verified diagonal-isotropy assumption.<n>nameA matches or outperforms AdamW and Muon under comparable tuning while using only momentum-level state.
  arXiv  Detail & Related papers  (2026-02-01T08:59:45Z)
• Transformers as Support Vector Machines [54.642793677472724]
  We establish a formal equivalence between the optimization geometry of self-attention and a hard-margin SVM problem. We characterize the implicit bias of 1-layer transformers optimized with gradient descent. We believe these findings inspire the interpretation of transformers as a hierarchy of SVMs that separates and selects optimal tokens.
  arXiv  Detail & Related papers  (2023-08-31T17:57:50Z)
• Transformers meet Stochastic Block Models: Attention with Data-Adaptive Sparsity and Cost [53.746169882193456]
  Recent works have proposed various sparse attention modules to overcome the quadratic cost of self-attention. We propose a model that resolves both problems by endowing each attention head with a mixed-membership Block Model. Our model outperforms previous efficient variants as well as the original Transformer with full attention.
  arXiv  Detail & Related papers  (2022-10-27T15:30:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.