Riemannian Attention Mechanisms for Transformers: A Theoretical Framework and Architecture Design

• URL: http://arxiv.org/abs/2608.01283v1
• Date: Sun, 02 Aug 2026 14:47:16 GMT
• Title: Riemannian Attention Mechanisms for Transformers: A Theoretical Framework and Architecture Design
• Abstract summary: All Transformer-based large language models compute attention via the Euclidean inner product.<n>We develop a theoretical framework that targets this structural limitation at the mathematical level.
• Score: 5.026698725861711
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: All Transformer-based large language models compute attention via the Euclidean inner product, an architectural choice that Dong et al. (2021) proved causes representational rank to decay doubly exponentially with depth in pure self-attention stacks. We develop a theoretical framework that targets this structural limitation at the mathematical level by replacing the flat Euclidean metric with learned per-token Riemannian metrics. Our contributions are threefold. (1) We prove that Riemannian attention scores with heterogeneous per-token metrics are non-Gram---they cannot be factorized as QK^T with factorization dimension O(d). We are explicit that this is a structural observation, not a proof of rank preservation. (2) We establish that low-rank metric factors render all geometric operations tractable: geodesic distance in O(d*r) per token and metric inversion in O(d*r^2) via the Woodbury identity---both far below the O(d^3) cost of a general matrix---making Riemannian attention feasible at billion-parameter scale with negligible overhead. (3) We present the Fiber Bundle Transformer, a complete architecture specification in which each token position carries its own Riemannian metric, attention is geodesic distance computation, feed-forward updates use metric-preconditioned steps, and the connection carries explicit curvature and torsion proxies. We derive formal predictions about correctly implemented geometric architectures and identify the central open problem: proving or disproving that heterogeneous Riemannian metrics prevent the rank collapse that row-stochastic attention matrices otherwise cause. This paper presents theoretical analysis and architectural design; empirical validation is the subject of future work.

Related papers

• Nested Inductive Bias Framework for SPD Manifold Learning [0.0]
  In Geometric Deep Learning, inductive biases serve two primary functions: enforcing manifold constraints and embedding relational priors.<n>We introduce a Nested Inductive Bias framework to pull back non-Euclidean target geometries onto the SPD manifold.<n>We also propose the Rational Conformal Metric to establish state-of-the-art robustness against outliers.
  arXiv  Detail & Related papers  (2026-09-03T20:50:18Z)
• Riemannian geometry meets fMRI: the advantages of modeling correlation manifolds and eigenvector subspaces [0.0]
  Correlation matrices are fundamental summaries of functional brain networks.<n>Existing geometric methods frequently lack closed-form operations or depend on arbitrary region ordering.<n>We introduce a scalable geometric framework with two components: the Off-log metric and Grassmannian subspace discrimination.
  arXiv  Detail & Related papers  (2026-05-21T11:22:31Z)
• NOETHER: A Constructive Framework for Metamorphic Pattern Discovery from Operator Algebras [6.691613382732758]
  Metamorphic Testing is recognised in IEEE/ISO software-testing standards and increasingly recommended for AI systems.<n>Existing approaches share an inductive grounding that leaves three foundational questions open.<n>We propose a framework whose downstream step from operator algebra to MetaPattern set is mechanical and provable.
  arXiv  Detail & Related papers  (2026-05-17T11:18:35Z)
• Riemannian Flow Matching for Disentangled Graph Domain Adaptation [51.98961391065951]
  Graph Domain Adaptation (GDA) typically uses adversarial learning to align graph embeddings in Euclidean space.<n>DisRFM is a geometry-aware GDA framework that unifies embedding and flow-based transport.
  arXiv  Detail & Related papers  (2026-01-31T11:05:35Z)
• Constant Metric Scaling in Riemannian Computation [0.31727619150610836]
  We show how constant metric scaling can be introduced without altering the geometric structures on which these methods rely.<n>The goal of this note is purely expository and is intended to clarify how a global metric scale parameter can be introduced without altering the geometric structures on which these methods rely.
  arXiv  Detail & Related papers  (2026-01-16T04:54:23Z)
• Riemannian Zeroth-Order Gradient Estimation with Structure-Preserving Metrics for Geodesically Incomplete Manifolds [57.179679246370114]
  We construct metrics that are geodesically complete while ensuring that every stationary point under the new metric remains stationary under the original one.<n>An $$-stationary point under the constructed metric $g'$ also corresponds to an $$-stationary point under the original metric $g'$.<n>Experiments on a practical mesh optimization task demonstrate that our framework maintains stable convergence even in the absence of geodesic completeness.
  arXiv  Detail & Related papers  (2026-01-12T22:08:03Z)
• Follow the Energy, Find the Path: Riemannian Metrics from Energy-Based Models [63.331590876872944]
  We propose a method for deriving Riemannian metrics directly from pretrained Energy-Based Models.<n>These metrics define spatially varying distances, enabling the computation of geodesics.<n>We show that EBM-derived metrics consistently outperform established baselines.
  arXiv  Detail & Related papers  (2025-05-23T12:18:08Z)
• Intrinsic Bayesian Cramér-Rao Bound with an Application to Covariance Matrix Estimation [49.67011673289242]
  This paper presents a new performance bound for estimation problems where the parameter to estimate lies in a smooth manifold. It induces a geometry for the parameter manifold, as well as an intrinsic notion of the estimation error measure.
  arXiv  Detail & Related papers  (2023-11-08T15:17:13Z)
• Identifying latent distances with Finslerian geometry [6.0188611984807245]
  Generative models cause the data space and the geodesics to be at best impractical, and at worst impossible to manipulate. In this work, we propose another metric whose geodesics explicitly minimise the expected length of the pullback metric. In high dimensions, we prove that both metrics converge to each other at a rate of $Oleft(frac1Dright)$.
  arXiv  Detail & Related papers  (2022-12-20T05:57:27Z)
• A Unifying and Canonical Description of Measure-Preserving Diffusions [60.59592461429012]
  A complete recipe of measure-preserving diffusions in Euclidean space was recently derived unifying several MCMC algorithms into a single framework. We develop a geometric theory that improves and generalises this construction to any manifold.
  arXiv  Detail & Related papers  (2021-05-06T17:36:55Z)
• Disentangling by Subspace Diffusion [72.1895236605335]
  We show that fully unsupervised factorization of a data manifold is possible if the true metric of the manifold is known. Our work reduces the question of whether unsupervised metric learning is possible, providing a unifying insight into the geometric nature of representation learning.
  arXiv  Detail & Related papers  (2020-06-23T13:33:19Z)
• An Inductive Bias for Distances: Neural Nets that Respect the Triangle Inequality [38.95108641775682]
  Distances are pervasive in machine learning and serve as similarity measures, loss functions, and learning targets. Deep metric learning architectures that respect the triangle inequality rely, almost exclusively, on Euclidean distance in the latent space. We show that our architectures outperform existing metric approaches when modeling graph distances and have a better inductive bias than non-metric approaches when training data is limited.
  arXiv  Detail & Related papers  (2020-02-14T00:47:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.