Generalizable Insights for Graph Transformers in Theory and Practice

• URL: http://arxiv.org/abs/2511.08028v1
• Date: Wed, 12 Nov 2025 01:35:00 GMT
• Title: Generalizable Insights for Graph Transformers in Theory and Practice
• Abstract summary: Graph Transformers (GTs) have shown strong empirical performance.<n>Current architectures vary widely in their use of attention mechanisms, positional embeddings (PEs), and expressivity.<n>Here, we propose a GT architecture using standard attention that incorporates many advancements for GTs from recent years.
• Score: 10.811483927795921
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Graph Transformers (GTs) have shown strong empirical performance, yet current architectures vary widely in their use of attention mechanisms, positional embeddings (PEs), and expressivity. Existing expressivity results are often tied to specific design choices and lack comprehensive empirical validation on large-scale data. This leaves a gap between theory and practice, preventing generalizable insights that exceed particular application domains. Here, we propose the Generalized-Distance Transformer (GDT), a GT architecture using standard attention that incorporates many advancements for GTs from recent years, and develop a fine-grained understanding of the GDT's representation power in terms of attention and PEs. Through extensive experiments, we identify design choices that consistently perform well across various applications, tasks, and model scales, demonstrating strong performance in a few-shot transfer setting without fine-tuning. Our evaluation covers over eight million graphs with roughly 270M tokens across diverse domains, including image-based object detection, molecular property prediction, code summarization, and out-of-distribution algorithmic reasoning. We distill our theoretical and practical findings into several generalizable insights about effective GT design, training, and inference.

Related papers

• Rethinking Feature Alignment in Generalist Graph Anomaly Detection: A Relational Fingerprint-based Approach [60.199985286210904]
  Generalist graph anomaly detection (GAD) aims to detect anomalies on unseen graphs without graph-specific retraining.<n>Existing approaches primarily focus on aligning heterogeneous data across domains, which ignores different feature semantics.<n>We propose Fingerprint-based generalist GAD approach (ReFi-GAD)<n>We show that ReFi-GAD significantly outperforms state-of-the-art methods in experiments on 14 datasets.
  arXiv  Detail & Related papers  (2026-05-25T05:12:26Z)
• Size Transferability of Graph Transformers with Convolutional Positional Encodings [82.27361992510494]
  Graph Transformers (GTs) are attention-based architectures for graph-structured data.<n>We study GTs through the lens of manifold limit models for graph sequences.<n>We show that GTs inherit transferability guarantees from their positional encodings.
  arXiv  Detail & Related papers  (2026-02-16T22:38:56Z)
• Plain Transformers are Surprisingly Powerful Link Predictors [57.01966734467712]
  Link prediction is a core challenge in graph machine learning, demanding models that capture rich and complex topological dependencies.<n>While Graph Neural Networks (GNNs) are the standard solution, state-of-the-art pipelines often rely on explicit structurals or memory-intensive node embeddings.<n>We present PENCIL, an encoder-only plain Transformer that replaces hand-crafted priors with attention over sampled local subgraphs.
  arXiv  Detail & Related papers  (2026-02-02T02:45:52Z)
• DiG: Differential Grounding for Enhancing Fine-Grained Perception in Multimodal Large Language Model [22.28268642142352]
  DiG (Differential Grounding) is a novel proxy task framework where MLLMs learn fine-grained perception by identifying and localizing all differences between similar image pairs without prior knowledge of their number.<n>Our results highlight differential grounding as a scalable and robust approach for advancing fine-grained visual reasoning in MLLMs.
  arXiv  Detail & Related papers  (2025-12-14T10:40:27Z)
• Exploring Graph-Transformer Out-of-Distribution Generalization Abilities [2.4063592468412276]
  Graph-transformer (GT) backbones have recently outperformed traditional message-passing neural networks (MPNNs) in multiple in-distribution (ID) benchmarks.<n>In this work, we address the challenge of out-of-distribution (OOD) generalization for graph neural networks.<n>We adapt several leading domain generalization (DG) algorithms to work with GTs and assess their performance on a benchmark designed to test a variety of distribution shifts.
  arXiv  Detail & Related papers  (2025-06-25T16:09:24Z)
• OpenGT: A Comprehensive Benchmark For Graph Transformers [13.214504021335749]
  Graph Transformers (GTs) have recently demonstrated remarkable performance across diverse domains.<n>This paper introduces OpenGT, a comprehensive benchmark for Graph Transformers.
  arXiv  Detail & Related papers  (2025-06-05T08:48:46Z)
• Towards A Universal Graph Structural Encoder [18.609281566514017]
  GFSE is a universal graph structural encoder designed to capture transferable structural patterns across diverse domains.<n>Built on a Graph Transformer, GFSE incorporates attention mechanisms informed by graph inductive bias, enabling it to encode intricate multi-level and fine-grained topological features.<n>GFSE achieves state-of-the-art performance in 81.6% evaluated cases, spanning diverse graph models and datasets.
  arXiv  Detail & Related papers  (2025-04-15T06:57:26Z)
• QT-DoG: Quantization-aware Training for Domain Generalization [58.439816306817306]
  We propose Quantization-aware Training for Domain Generalization (QT-DoG)<n>We demonstrate that weight quantization effectively leads to flatter minima in the loss landscape.<n> QT-DoG exploits quantization as an implicit regularizer by inducing noise in model weights.
  arXiv  Detail & Related papers  (2024-10-08T13:21:48Z)
• What Improves the Generalization of Graph Transformers? A Theoretical Dive into the Self-attention and Positional Encoding [67.59552859593985]
  Graph Transformers, which incorporate self-attention and positional encoding, have emerged as a powerful architecture for various graph learning tasks. This paper introduces first theoretical investigation of a shallow Graph Transformer for semi-supervised classification.
  arXiv  Detail & Related papers  (2024-06-04T05:30:16Z)
• GiT: Towards Generalist Vision Transformer through Universal Language Interface [94.33443158125186]
  This paper proposes a simple, yet effective framework, called GiT, simultaneously applicable for various vision tasks only with a vanilla ViT. GiT is a multi-task visual model, jointly trained across five representative benchmarks without task-specific fine-tuning.
  arXiv  Detail & Related papers  (2024-03-14T13:47:41Z)
• What Matters When Repurposing Diffusion Models for General Dense Perception Tasks? [49.84679952948808]
  Recent works show promising results by simply fine-tuning T2I diffusion models for dense perception tasks.<n>We conduct a thorough investigation into critical factors that affect transfer efficiency and performance when using diffusion priors.<n>Our work culminates in the development of GenPercept, an effective deterministic one-step fine-tuning paradigm tailed for dense visual perception tasks.
  arXiv  Detail & Related papers  (2024-03-10T04:23:24Z)
• Exploring Sparsity in Graph Transformers [67.48149404841925]
  Graph Transformers (GTs) have achieved impressive results on various graph-related tasks. However, the huge computational cost of GTs hinders their deployment and application, especially in resource-constrained environments. We propose a comprehensive textbfGraph textbfTransformer textbfSParsification (GTSP) framework that helps to reduce the computational complexity of GTs.
  arXiv  Detail & Related papers  (2023-12-09T06:21:44Z)
• The Neglected Sibling: Isotropic Gaussian Posterior for VAE [5.377898309773304]
  We propose a simple modification to Variational Autoencoders (VAEs) by using an Isotropic Gaussian Posterior (IGP) This model avoids the sub-optimal behavior of VAEs related to inactive dimensions in the representation space.
  arXiv  Detail & Related papers  (2021-10-14T14:12:52Z)
• Node Masking: Making Graph Neural Networks Generalize and Scale Better [71.51292866945471]
  Graph Neural Networks (GNNs) have received a lot of interest in the recent times. In this paper, we utilize some theoretical tools to better visualize the operations performed by state of the art spatial GNNs. We introduce a simple concept, Node Masking, that allows them to generalize and scale better.
  arXiv  Detail & Related papers  (2020-01-17T06:26:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.