Pure Transformers are Powerful Graph Learners

• URL: http://arxiv.org/abs/2207.02505v1
• Date: Wed, 6 Jul 2022 08:13:06 GMT
• Title: Pure Transformers are Powerful Graph Learners
• Authors: Jinwoo Kim, Tien Dat Nguyen, Seonwoo Min, Sungjun Cho, Moontae Lee, Honglak Lee, Seunghoon Hong
• Abstract summary: We show that standard Transformers without graph-specific modifications can lead to promising results in graph learning both in theory and practice. We prove that this approach is theoretically at least as expressive as an invariant graph network (2-IGN) composed of equivariant linear layers. Our method coined Tokenized Graph Transformer (TokenGT) achieves significantly better results compared to GNN baselines and competitive results.
• Score: 51.36884247453605
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We show that standard Transformers without graph-specific modifications can lead to promising results in graph learning both in theory and practice. Given a graph, we simply treat all nodes and edges as independent tokens, augment them with token embeddings, and feed them to a Transformer. With an appropriate choice of token embeddings, we prove that this approach is theoretically at least as expressive as an invariant graph network (2-IGN) composed of equivariant linear layers, which is already more expressive than all message-passing Graph Neural Networks (GNN). When trained on a large-scale graph dataset (PCQM4Mv2), our method coined Tokenized Graph Transformer (TokenGT) achieves significantly better results compared to GNN baselines and competitive results compared to Transformer variants with sophisticated graph-specific inductive bias. Our implementation is available at https://github.com/jw9730/tokengt.

Related papers

• Learning Graph Quantized Tokenizers for Transformers [28.79505338383552]
  Graph Transformers (GTs) have emerged as a leading model in deep learning, outperforming Graph Neural Networks (GNNs) in various graph learning tasks. We introduce GQT (textbfGraph textbfQuantized textbfTokenizer), which decouples tokenizer training from Transformer training by leveraging graph self-supervised learning. By combining the GQT with token modulation, a Transformer encoder achieves state-of-the-art performance on 16 out of 18 benchmarks, including large-scale homophilic and heterophilic datasets.
  arXiv  Detail & Related papers  (2024-10-17T17:38:24Z)
• SpikeGraphormer: A High-Performance Graph Transformer with Spiking Graph Attention [1.4126245676224705]
  Graph Transformers have emerged as a promising solution to alleviate the inherent limitations of Graph Neural Networks (GNNs) We propose a novel insight into integrating SNNs with Graph Transformers and design a Spiking Graph Attention (SGA) module. SpikeGraphormer consistently outperforms existing state-of-the-art approaches across various datasets.
  arXiv  Detail & Related papers  (2024-03-21T03:11:53Z)
• Graph Inductive Biases in Transformers without Message Passing [47.238185813842996]
  New Graph Inductive bias Transformer (GRIT) incorporates graph inductive biases without using message passing. GRIT achieves state-of-the-art empirical performance across a variety of graph datasets.
  arXiv  Detail & Related papers  (2023-05-27T22:26:27Z)
• Graph Propagation Transformer for Graph Representation Learning [36.01189696668657]
  We propose a new attention mechanism called Graph Propagation Attention (GPA) It explicitly passes the information among nodes and edges in three ways, i.e. node-to-node, node-to-edge, and edge-to-node. We show that our method outperforms many state-of-the-art transformer-based graph models with better performance.
  arXiv  Detail & Related papers  (2023-05-19T04:42:58Z)
• AGFormer: Efficient Graph Representation with Anchor-Graph Transformer [95.1825252182316]
  We propose a novel graph Transformer architecture, termed Anchor Graph Transformer (AGFormer) AGFormer first obtains some representative anchors and then converts node-to-node message passing into anchor-to-anchor and anchor-to-node message passing process. Extensive experiments on several benchmark datasets demonstrate the effectiveness and benefits of proposed AGFormer.
  arXiv  Detail & Related papers  (2023-05-12T14:35:42Z)
• PatchGT: Transformer over Non-trainable Clusters for Learning Graph Representations [18.203910156450085]
  We propose a new Transformer-based graph neural network: Patch Graph Transformer (PatchGT) Unlike previous transformer-based models for learning graph representations, PatchGT learns from non-trainable graph patches, not from nodes directly. PatchGT achieves higher than 1-WL-type GNNs, and the empirical study shows that PatchGT achieves competitive performances on benchmark datasets.
  arXiv  Detail & Related papers  (2022-11-26T01:17:23Z)
• Representation Power of Graph Neural Networks: Improved Expressivity via Algebraic Analysis [124.97061497512804]
  We show that standard Graph Neural Networks (GNNs) produce more discriminative representations than the Weisfeiler-Lehman (WL) algorithm. We also show that simple convolutional architectures with white inputs, produce equivariant features that count the closed paths in the graph.
  arXiv  Detail & Related papers  (2022-05-19T18:40:25Z)
• Do Transformers Really Perform Bad for Graph Representation? [62.68420868623308]
  We present Graphormer, which is built upon the standard Transformer architecture. Our key insight to utilizing Transformer in the graph is the necessity of effectively encoding the structural information of a graph into the model.
  arXiv  Detail & Related papers  (2021-06-09T17:18:52Z)
• Graph Contrastive Learning with Augmentations [109.23158429991298]
  We propose a graph contrastive learning (GraphCL) framework for learning unsupervised representations of graph data. We show that our framework can produce graph representations of similar or better generalizability, transferrability, and robustness compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-10-22T20:13:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.