Related papers: Attention-based PCA

Attention-based PCA

URL: http://arxiv.org/abs/2605.18315v1
Date: Mon, 18 May 2026 12:34:00 GMT
Title: Attention-based PCA
Abstract summary: We study attention mechanisms through the lens of a canonical unsupervised problem: principal component analysis (PCA)<n>Our analysis covers both finite and infinite prompt regimes. In the infinite-prompt limit, we prove convergence to globally optimal solutions aligned with the leading spectral direction, while in the finiteprompt setting we show that the same behavior emerges up to sampling effects.<n>These results demonstrate that attention inherently performs PCA-like computations under unsupervised objectives, providing a theoretical foundation for its representation-learning capabilities.
Score: 6.467090475885798
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Abstract: We study attention mechanisms through the lens of a canonical unsupervised problem: principal component analysis (PCA). We show that, when trained on Gaussian data, both softmax and linear attention layers learn parameters that align with the principal eigenvectors of the covariance matrix, thereby establishing a direct and explicit connection with PCA. Our analysis covers both finite and infinite prompt regimes. In the infinite-prompt limit, we prove convergence to globally optimal solutions aligned with the leading spectral direction, while in the finiteprompt setting we show that the same behavior emerges up to sampling effects. We further extend the analysis to an in-context setting with spiked Wishart covariances, where attention successfully recovers the underlying signal direction. These results demonstrate that attention inherently performs PCA-like computations under unsupervised objectives, providing a theoretical foundation for its representation-learning capabilities.

Related papers

PAC-Bayesian Adversarially Robust Generalization for Message Passing Graph Neural Networks: A Sensitivity Analysis [80.64607823130437]
We extend PAC-Bayesian framework from deep neural networks to message passing GNNs.<n>We derive a tighter robust generalization bound in the adversarial setting.
arXiv Detail & Related papers (2026-06-04T15:31:21Z)
Curvature-Aware PCA with Geodesic Tangent Space Aggregation for Semi-Supervised Learning [52.452902154360565]
GTSA-PCA is a geometric extension of Principal Component Analysis.<n>It integrates curvature awareness and geodesic consistency within a unified spectral framework.<n>Our results position GTSA-PCA as a principled bridge between statistical and geometric approaches to dimensionality reduction.
arXiv Detail & Related papers (2026-04-20T20:36:36Z)
On Multi-Step Theorem Prediction via Non-Parametric Structural Priors [50.16583672681106]
In this work, we explore training-free theorem prediction through the lens of in-context learning (ICL)<n>We propose Theorem Precedence Graphs, which encode temporal dependencies from historical solution traces as directed graphs, and impose explicit topological constraints that effectively prune the search space during inference.<n>Experiments on the FormalGeo7k benchmark show that our method achieves 89.29% accuracy, substantially outperforming ICL baselines and matching state-of-the-art supervised models.
arXiv Detail & Related papers (2026-03-05T06:08:50Z)
Sycophancy Hides Linearly in the Attention Heads [26.373090085794036]
correct-to-incorrect sycophancy signals are most linearly separable within multi-head attention activations.<n>We train linear probes across the residual stream, multilayer perceptron (MLP), and attention layers to analyze where these signals emerge.
arXiv Detail & Related papers (2026-01-23T11:04:04Z)
Towards A Unified PAC-Bayesian Framework for Norm-based Generalization Bounds [63.47271262149291]
We propose a unified framework for PAC-Bayesian norm-based generalization.<n>The key to our approach is a sensitivity matrix that quantifies the network outputs with respect to structured weight perturbations.<n>We derive a family of generalization bounds that recover several existing PAC-Bayesian results as special cases.
arXiv Detail & Related papers (2026-01-13T00:42:22Z)
Hallucination Detection in LLMs with Topological Divergence on Attention Graphs [60.83579255387347]
Hallucination, i.e., generating factually incorrect content, remains a critical challenge for large language models.<n>We introduce TOHA, a TOpology-based HAllucination detector in the RAG setting.
arXiv Detail & Related papers (2025-04-14T10:06:27Z)
Do algorithms and barriers for sparse principal component analysis extend to other structured settings? [9.339550283840769]
We study a principal component analysis problem under the spiked Wishart model. We establish fundamental limits that depend on the geometry of the problem instance. We show that a natural projected power method exhibits local convergence to the statistically near-optimal neighborhood of the solution.
arXiv Detail & Related papers (2023-07-25T14:30:00Z)
Contrastive Learning Can Find An Optimal Basis For Approximately View-Invariant Functions [18.440569330385323]
We show that multiple contrastive learning methods can be reinterpreted as learning kernel functions that approximate a fixed positive-pair kernel. We prove that a simple representation obtained by combining this kernel with PCA provably minimizes the worst-case approximation error of linear predictors.
arXiv Detail & Related papers (2022-10-04T20:02:52Z)
Supervised PCA: A Multiobjective Approach [70.99924195791532]
Methods for supervised principal component analysis (SPCA) We propose a new method for SPCA that addresses both of these objectives jointly. Our approach accommodates arbitrary supervised learning losses and, through a statistical reformulation, provides a novel low-rank extension of generalized linear models.
arXiv Detail & Related papers (2020-11-10T18:46:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.