Self-Attention as Transport: Limits of Symmetric Spectral Diagnostics

• URL: http://arxiv.org/abs/2605.04893v1
• Date: Wed, 06 May 2026 13:25:13 GMT
• Title: Self-Attention as Transport: Limits of Symmetric Spectral Diagnostics
• Abstract summary: We show that uniform causal attention satisfies an $n$-independent floor $ge 1/5$ with worst cut at $tast/n approx 0.32$, while window attention pierces the floor as $O(w/n)$.<n>The resulting two-axis diagnostic yields a falsifiable polarity prediction.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models hallucinate in predictable ways: attention routing fails by over-concentrating on a narrow set of positions, or by spreading so diffusely that relevance is diluted, and the shape of the failure carries diagnostic signal. A widely used family of spectral methods analyzes the symmetric component of the degree-normalized attention operator, which governs transport capacity; we prove that every transpose-invariant spectral diagnostic of this operator is structurally orientation-blind (it cannot distinguish an operator from its transpose, and therefore cannot detect information-flow direction), with a quantitative converse establishing the asymmetry coefficient $G$ as the unique control parameter for direction. Pairing this with a closed-form bipartite-Cheeger landscape for canonical causal architectures, we show that uniform causal attention satisfies an $n$-independent floor $φ\ge 1/5$ with worst cut at $t^\ast/n \approx 0.32$, while window attention pierces the floor as $O(w/n)$; failure modes are shape-different, not just value-different. The resulting two-axis diagnostic ($φ$ for capacity, $G$ for direction) yields a falsifiable polarity prediction: bottleneck- and diffuse-dominated benchmarks should exhibit opposite polarity. Under length-controlled evaluation, transport features retain interpretable signal (LC-AUROC from 0.62 to 0.84) on tested models up to 8B parameters, with polarity reversing as predicted between HaluEval and MedHallu.

Related papers

• $\ ext{GSF-}χ$: Global Stereochemical Fields for Chiral Graph Transformers [49.85093436361195]
  We introduce GSF-$$, a graph transformer in which stereogenic units modulate all pairwise interactions.<n>A $C$ projection separates mirror-even ECD peak counts and positions from mirror-odd peak signs.<n>GSF-$$ leads every central-ECD output and improves axial Rotation and Symbol by $12.6%$ and $7.9%$ over the strongest baseline.
  arXiv  Detail & Related papers  (2026-09-11T07:41:20Z)
• Cone Extended Rayleigh Quotients for Directed Graph Learning: Minimax Spectral Certificates, Sensitivity, and Adaptive Control [0.0]
  We develop a learning-oriented methodology for spectral certification, sensitivity analysis, and control without requiring symmetry, nonnegativity, or cone preservation.<n>For a simple interior level, the right and left modes satisfy [ D_C(B)[H]=v_CT H u_C, ] yielding first-order optimal graph-supported interventions.<n> Numerical experiments demonstrate the applicability of the approach beyond cone-preserving operators and in directed learning settings.
  arXiv  Detail & Related papers  (2026-08-27T13:37:02Z)
• Intrinsic Structure: Spectral Identifiability for Mechanistic Interpretability [51.56484100374058]
  We prove the first identifiability theorem for a mechanistic-interpretability primitive.<n>The spectrum is an identifiable, model-intrinsic fingerprint with a stated error bar, not a legible decomposition.
  arXiv  Detail & Related papers  (2026-08-10T19:42:01Z)
• Layer-Resolved Optimal Transport for Hallucination Detection in NMT and Abstractive Summarization [0.0]
  Optimal transport (OT) has been shown to detect hallucinations in neural machine translation (NMT)<n>We extend this analysis to all six decoder layers of the Fairseq DE-EN model.
  arXiv  Detail & Related papers  (2026-06-11T11:30:08Z)
• Predictive Maps of Multi-Agent Reasoning: A Successor-Representation Spectrum for LLM Communication Topologies [0.0]
  Existing evaluation answers these questions only post hoc and only for the task measured.<n>We introduce a structural diagnostic for multi-agent LLM communication graphs based on the successor representation $M = (I - P)-1$ of the row-stochastic communication operator.<n>We derive closed-form spectra for the chain, star, and mesh under row-stochastic normalization, and validate the predictions on a 12-step structured state-tracking task.
  arXiv  Detail & Related papers  (2026-05-12T03:11:39Z)
• Cross-Spectral Witness for Hidden Nonequilibrium Beyond the Scalar Ceiling [18.923595971721344]
  coarse graining may absorb hidden forcing into an apparently equilibrium-like reduced description.<n>For scalar observables of linear systems, no time-irreversibility statistic can detect the underlying drive.<n>Within general CSM this certifies shared hidden-sector drive.
  arXiv  Detail & Related papers  (2026-04-04T15:54:29Z)
• Stability and Generalization of Push-Sum Based Decentralized Optimization over Directed Graphs [55.77845440440496]
  Push-based decentralized communication enables optimization over communication networks, where information exchange may be asymmetric.<n>We develop a unified uniform-stability framework for the Gradient Push (SGP) algorithm.<n>A key technical ingredient is an imbalance-aware generalization bound through two quantities.
  arXiv  Detail & Related papers  (2026-02-24T05:32:03Z)
• Revisiting Zeroth-Order Optimization: Minimum-Variance Two-Point Estimators and Directionally Aligned Perturbations [57.179679246370114]
  We identify the distribution of random perturbations that minimizes the estimator's variance as the perturbation stepsize tends to zero.<n>Our findings reveal that such desired perturbations can align directionally with the true gradient, instead of maintaining a fixed length.
  arXiv  Detail & Related papers  (2025-10-22T19:06:39Z)
• Transformers as Support Vector Machines [54.642793677472724]
  We establish a formal equivalence between the optimization geometry of self-attention and a hard-margin SVM problem. We characterize the implicit bias of 1-layer transformers optimized with gradient descent. We believe these findings inspire the interpretation of transformers as a hierarchy of SVMs that separates and selects optimal tokens.
  arXiv  Detail & Related papers  (2023-08-31T17:57:50Z)
• Neural-network quantum state study of the long-range antiferromagnetic Ising chain [0.771303749110121]
  We investigate quantum phase transitions in the transverse field Ising chain with algebraically decaying long-range (LR) antiferromagnetic interactions. We find that the universal ratio of the SR limit does not hold for $alpha_mathrmLR 2$, implying a deviation in the criticality.
  arXiv  Detail & Related papers  (2023-08-18T17:58:36Z)
• Analytic Signal Phase in $N-D$ by Linear Symmetry Tensor--fingerprint modeling [69.35569554213679]
  We show that the Analytic Signal phase, and its gradient have a hitherto unstudied discontinuity in $2-D $ and higher dimensions. This shortcoming can result in severe artifacts whereas the problem does not exist in $1-D $ signals. We suggest the use of Linear Symmetry phase, relying on more than one set of Gabor filters, but with a negligible computational add-on.
  arXiv  Detail & Related papers  (2020-05-16T21:17:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.