Function graph transformers universally approximate operators between function spaces

• URL: http://arxiv.org/abs/2605.17968v1
• Date: Mon, 18 May 2026 07:25:14 GMT
• Title: Function graph transformers universally approximate operators between function spaces
• Abstract summary: We study the approximation of nonlinear operators between function spaces by transformers.<n>Our approach is to lift functions to measures supported on their graphs and leverage a recently introduced measure-theoretic view of transformers.<n>We show that this framework models discretization refinement via convergence measures and provides a natural setting for operator learning.
• Score: 14.886643480836817
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We study the approximation of nonlinear operators between function spaces by transformers. Our approach is to lift functions to measures supported on their graphs and leverage a recently introduced measure-theoretic view of transformers. A function $h$ is represented by its graph measure $γ_h$, with finite tokens $\{(x_j,h(x_j))\}_{j=1}^N$ being its empirical approximations. We show that this framework elegantly models discretization refinement via convergence of measures and provides a natural setting for operator learning. Within this framework, we introduce function graph transformers, a graph-preserving subclass of measure-theoretic transformers that maps graph measures to graph measures, which is to say that outputs remain single-valued functions. Crucially, this additional structure does not reduce generality: we prove that the resulting graph-preserving maps can be approximated by finite compositions of standard softmax self-attention layers and pointwise MLPs, yielding universal approximation results for broad classes of nonlinear operators. Unlike existing theoretical approaches to operator learning with transformers, the measure-theoretic framework also accommodates regularized negative-order Sobolev inputs for which discretization invariance is particularly challenging, as well as query points on different output domains. Overall, function graph transformers provide a continuum viewpoint and mathematical toolkit for transformer-based operator learning, clarifying the roles of positional encodings, graph structure, regularization, and ensuring consistency across discretizations.

Related papers

• From Self-Attention to Connection Laplacian: A Unified Operator View of Transformers [53.505136935517925]
  Self-attention is a ubiquitous primitive in modern sequence models, yet its operator-level geometry is only partially understood.<n>We prove that single-head attention (SHA) is exactly a connection propagation step with constant transport.<n>We find that trained Transformers across scales (from 124M to 8B) and structures (encoder/decoder) exhibit geometric structure consistent with our theory.
  arXiv  Detail & Related papers  (2026-07-12T09:44:54Z)
• TensorLens: End-to-End Transformer Analysis via High-Order Attention Tensors [53.891337639229285]
  We introduce attentionLens, a novel formulation that captures the entire transformer as a single, input-dependent linear operator expressed through a high-order attention-interaction connection.<n>Our experiments demonstrate that the attention tensor can serve as a powerful foundation for developing tools aimed at interpretability and model understanding.
  arXiv  Detail & Related papers  (2026-01-25T19:21:25Z)
• Graded Transformers [0.0]
  We introduce the Graded Transformer framework, a new class of sequence models that embed inductive biases through grading on vector spaces.<n>The framework enables adaptive feature prioritization, overcoming limitations of fixed grades in earlier models.<n>The Graded Transformer provides a mathematically principled approach to hierarchical learning and neuro-symbolic reasoning.
  arXiv  Detail & Related papers  (2025-07-27T02:34:08Z)
• GITO: Graph-Informed Transformer Operator for Learning Complex Partial Differential Equations [0.0]
  We present a novel graph-informed transformer operator (GITO) architecture for learning complex partial differential equation systems.<n>GITO consists of two main modules: a hybrid graph transformer (HGT) and a transformer neural operator (TNO)<n> Empirical results on benchmark PDE tasks demonstrate that GITO outperforms existing transformer-based neural operators.
  arXiv  Detail & Related papers  (2025-06-16T18:35:45Z)
• Transformers Meet In-Context Learning: A Universal Approximation Theory [25.513848079509653]
  We develop a universal approximation theory to elucidate how transformers enable in-context learning.<n>For a general class of functions, we demonstrate how to construct a transformer that can predict based on a few noisy in-context examples.
  arXiv  Detail & Related papers  (2025-06-05T16:12:51Z)
• Graph Laplacian Wavelet Transformer via Learnable Spectral Decomposition [0.0]
  Existing sequence to sequence models for structured language tasks rely heavily on the dot product self attention mechanism.<n>We introduce the Graph Wavelet Transformer (GWT), a novel architecture that replaces this bottleneck with a learnable, multi scale wavelet transform.
  arXiv  Detail & Related papers  (2025-05-09T00:13:23Z)
• Plain Transformers Can be Powerful Graph Learners [64.50059165186701]
  Researchers have attempted to migrate Transformers to graph learning, but most advanced Graph Transformers have strayed far from plain Transformers.<n>This work demonstrates that the plain Transformer architecture can be a powerful graph learner.
  arXiv  Detail & Related papers  (2025-04-17T02:06:50Z)
• SGFormer: Single-Layer Graph Transformers with Approximation-Free Linear Complexity [74.51827323742506]
  We evaluate the necessity of adopting multi-layer attentions in Transformers on graphs. We show that one-layer propagation can be reduced to one-layer propagation, with the same capability for representation learning. It suggests a new technical path for building powerful and efficient Transformers on graphs.
  arXiv  Detail & Related papers  (2024-09-13T17:37:34Z)
• What Improves the Generalization of Graph Transformers? A Theoretical Dive into the Self-attention and Positional Encoding [67.59552859593985]
  Graph Transformers, which incorporate self-attention and positional encoding, have emerged as a powerful architecture for various graph learning tasks. This paper introduces first theoretical investigation of a shallow Graph Transformer for semi-supervised classification.
  arXiv  Detail & Related papers  (2024-06-04T05:30:16Z)
• Beyond symmetrization: effective adjacency matrices and renormalization for (un)singed directed graphs [0.0]
  We define the concept of effective adjacency matrices that arise from the definition of deformed Laplacian operators. These effective matrices allow mapping generic graphs to a family of unsigned, undirected graphs. We show how the Hodge-Helmholtz decomposition can assist us in navigating this complexity.
  arXiv  Detail & Related papers  (2024-06-03T16:48:25Z)
• Curve Your Attention: Mixed-Curvature Transformers for Graph Representation Learning [77.1421343649344]
  We propose a generalization of Transformers towards operating entirely on the product of constant curvature spaces. We also provide a kernelized approach to non-Euclidean attention, which enables our model to run in time and memory cost linear to the number of nodes and edges.
  arXiv  Detail & Related papers  (2023-09-08T02:44:37Z)
• What Dense Graph Do You Need for Self-Attention? [73.82686008622596]
  We present Hypercube Transformer, a sparse Transformer that models token interactions in a hypercube and shows comparable or even better results with vanilla Transformer. Experiments on tasks requiring various sequence lengths lay validation for our graph function well.
  arXiv  Detail & Related papers  (2022-05-27T14:36:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.