Expressivity of Transformers: A Tropical Geometry Perspective

• URL: http://arxiv.org/abs/2604.14727v1
• Date: Thu, 16 Apr 2026 07:39:00 GMT
• Title: Expressivity of Transformers: A Tropical Geometry Perspective
• Abstract summary: We introduce a tropical geometry framework to characterize transformers' exact partitioning capabilities.<n>We prove it evaluates exactly to a Power Voronoi Diagram in the zero-temperature limit.<n> Importantly, we guarantee that this idealized polyhedral skeleton is geometrically stable.
• Score: 11.888882732753922
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: To quantify the geometric expressivity of transformers, we introduce a tropical geometry framework to characterize their exact spatial partitioning capabilities. By modeling self-attention as a vector-valued tropical rational map, we prove it evaluates exactly to a Power Voronoi Diagram in the zero-temperature limit. Building on this equivalence, we establish a combinatorial rationale for Multi-Head Self-Attention (MHSA): via the Minkowski sum of Newton polytopes, multi-head aggregation expands the polyhedral complexity to $\mathcal{O}(N^H)$, overcoming the $\mathcal{O}(N)$ bottleneck of single heads. Extending this to deep architectures, we derive the first tight asymptotic bounds on the number of linear regions in transformers ($Θ(N^{d_{\text{model}}L})$), demonstrating a combinatorial explosion driven intrinsically by sequence length $N$, ambient embedding dimension $d_{\text{model}}$, and network depth $L$. Importantly, we guarantee that this idealized polyhedral skeleton is geometrically stable: finite-temperature soft attention preserves these topological partitions via exponentially tight differential approximation bounds.

Related papers

• $\mathbb{SL}(n)$ Representation Learning: An Intrinsic Mixed-Curvature Space with Higher Curvature Capacities and Deeper Order-Aware Composition [47.80325814873361]
  Mixed-curvature representation learning seeks to capture rich geometric structures that cannot be adequately modeled by a single curvature regime.<n>We introduce the $mathbbSL(n)$ space, a representation geometry defined by the simple $det(A)variant=1$ constraint and a left in Schatten-$p$ structure.<n>Despite this minimal construction, $mathbbSL(n)$ exhibits pointwise negative, zero, and flag curvature around a common flagpole.
  arXiv  Detail & Related papers  (2026-09-14T05:52:14Z)
• The Geometry of Semantic Space: A Continuous Geometric Framework for the Transformer Architecture [0.7310043452300737]
  We present a continuous framework that models the discrete operations of the Transformer architecture as an integro-differential equation (IDE) on a fiber bundle.<n>We conduct a six-part experimental campaign spanning $124$M to $8$B parameters.<n>The results demonstrate that analyzing Transformers through the lens of differential geometry provides a descriptive vocabulary for large language models.
  arXiv  Detail & Related papers  (2026-07-19T09:08:51Z)
• Spectral and thermodynamic properties of supersymmetric quantum systems with self-adjoint deformed momentum [0.0]
  We build a rigorous framework for quantum systems with geometric deformations by constructing a strictly self-adjoint deformed momentum operator.<n>We provide the first complete thermodynamic characterization of such systems by analytically evaluating the partition function.<n>These results establish geometric deformation as a tunable parameter for engineering quantum thermodynamic responses in curved nanostructures.
  arXiv  Detail & Related papers  (2026-06-20T15:43:06Z)
• Intrinsic Wasserstein Rates for Score-Based Generative Models on Smooth Manifolds [61.14405512940818]
  We show that Score-based generative models are trained in high-dimensional ambient spaces.<n>We use a ReLU implementation of nearest-projection coordinates via finite intrinsic anchors and Gauss Newton.
  arXiv  Detail & Related papers  (2026-05-15T10:20:05Z)
• Riemannian Langevin Dynamics: Strong Convergence of Geometric Euler-Maruyama Scheme [51.56484100374058]
  Low-dimensional structure in real-world data plays an important role in the success of generative models.<n>We prove convergence theory of numerical schemes for manifold-valued differential equations.
  arXiv  Detail & Related papers  (2026-03-04T01:29:35Z)
• Latent Object Permanence: Topological Phase Transitions, Free-Energy Principles, and Renormalization Group Flows in Deep Transformer Manifolds [0.5729426778193398]
  We study the emergence of multi-step reasoning in deep Transformer language models through a geometric and statistical-physics lens.<n>We formalize the forward pass as a discrete coarse-graining map and relate the appearance of stable "concept basins" to fixed points of this renormalization-like dynamics.<n>The resulting low-entropy regime is characterized by a spectral tail collapse and by the formation of transient, reusable object-like structures in representation space.
  arXiv  Detail & Related papers  (2026-01-16T23:11:02Z)
• From Universal Approximation Theorem to Tropical Geometry of Multi-Layer Perceptrons [0.0]
  We revisit the Universal Approximation Theorem through the lens of the tropical geometry of neural networks.<n>We introduce constructive, geometry-aware architecture for sigmoidal multi-layer perceptrons.
  arXiv  Detail & Related papers  (2025-10-16T13:15:39Z)
• Expressive Power of Deep Networks on Manifolds: Simultaneous Approximation [2.815765641180636]
  We show that a constant-depth $mathrmReLUk-1$ network with bounded weights can approximate any function in the Sobolev space.<n>We also prove that our construction is nearly optimal by showing the required number of parameters matches up to a logarithmic factor.
  arXiv  Detail & Related papers  (2025-09-11T11:28:20Z)
• The Generative Leap: Sharp Sample Complexity for Efficiently Learning Gaussian Multi-Index Models [71.5283441529015]
  In this work we consider generic Gaussian Multi-index models, in which the labels only depend on the (Gaussian) $d$-dimensional inputs through their projection onto a low-dimensional $r = O_d(1)$ subspace.<n>We introduce the generative leap exponent $kstar$, a natural extension of the generative exponent from [Damian et al.'24] to the multi-index setting.
  arXiv  Detail & Related papers  (2025-06-05T18:34:56Z)
• Learning with Norm Constrained, Over-parameterized, Two-layer Neural Networks [54.177130905659155]
  Recent studies show that a reproducing kernel Hilbert space (RKHS) is not a suitable space to model functions by neural networks. In this paper, we study a suitable function space for over- parameterized two-layer neural networks with bounded norms.
  arXiv  Detail & Related papers  (2024-04-29T15:04:07Z)
• Transolver: A Fast Transformer Solver for PDEs on General Geometries [66.82060415622871]
  We present Transolver, which learns intrinsic physical states hidden behind discretized geometries. By calculating attention to physics-aware tokens encoded from slices, Transovler can effectively capture intricate physical correlations. Transolver achieves consistent state-of-the-art with 22% relative gain across six standard benchmarks and also excels in large-scale industrial simulations.
  arXiv  Detail & Related papers  (2024-02-04T06:37:38Z)
• Dist2Cycle: A Simplicial Neural Network for Homology Localization [66.15805004725809]
  Simplicial complexes can be viewed as high dimensional generalizations of graphs that explicitly encode multi-way ordered relations. We propose a graph convolutional model for learning functions parametrized by the $k$-homological features of simplicial complexes.
  arXiv  Detail & Related papers  (2021-10-28T14:59:41Z)
• A deep network construction that adapts to intrinsic dimensionality beyond the domain [79.23797234241471]
  We study the approximation of two-layer compositions $f(x) = g(phi(x))$ via deep networks with ReLU activation. We focus on two intuitive and practically relevant choices for $phi$: the projection onto a low-dimensional embedded submanifold and a distance to a collection of low-dimensional sets.
  arXiv  Detail & Related papers  (2020-08-06T09:50:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.