Edge-augmented Graph Transformers: Global Self-attention is Enough for Graphs

• URL: http://arxiv.org/abs/2108.03348v1
• Date: Sat, 7 Aug 2021 02:18:11 GMT
• Title: Edge-augmented Graph Transformers: Global Self-attention is Enough for Graphs
• Authors: Md Shamim Hussain, Mohammed J. Zaki and Dharmashankar Subramanian
• Abstract summary: We propose a simple yet powerful extension to the transformer - residual edge channels. The resultant framework, which we call Edge-augmented Graph Transformer (EGT), can directly accept, process and output structural information as well as node information. Our framework, which relies on global node feature aggregation, achieves better performance compared to Graph Convolutional Networks (GCN)
• Score: 24.796242917673755
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Transformer neural networks have achieved state-of-the-art results for unstructured data such as text and images but their adoption for graph-structured data has been limited. This is partly due to the difficulty in incorporating complex structural information in the basic transformer framework. We propose a simple yet powerful extension to the transformer - residual edge channels. The resultant framework, which we call Edge-augmented Graph Transformer (EGT), can directly accept, process and output structural information as well as node information. This simple addition allows us to use global self-attention, the key element of transformers, directly for graphs and comes with the benefit of long-range interaction among nodes. Moreover, the edge channels allow the structural information to evolve from layer to layer, and prediction tasks on edges can be derived directly from these channels. In addition to that, we introduce positional encodings based on Singular Value Decomposition which can improve the performance of EGT. Our framework, which relies on global node feature aggregation, achieves better performance compared to Graph Convolutional Networks (GCN), which rely on local feature aggregation within a neighborhood. We verify the performance of EGT in a supervised learning setting on a wide range of experiments on benchmark datasets. Our findings indicate that convolutional aggregation is not an essential inductive bias for graphs and global self-attention can serve as a flexible and adaptive alternative to graph convolution.

Related papers

• SGFormer: Single-Layer Graph Transformers with Approximation-Free Linear Complexity [74.51827323742506]
  We evaluate the necessity of adopting multi-layer attentions in Transformers on graphs. We show that one-layer propagation can be reduced to one-layer propagation, with the same capability for representation learning. It suggests a new technical path for building powerful and efficient Transformers on graphs.
  arXiv  Detail & Related papers  (2024-09-13T17:37:34Z)
• Learning to Model Graph Structural Information on MLPs via Graph Structure Self-Contrasting [50.181824673039436]
  We propose a Graph Structure Self-Contrasting (GSSC) framework that learns graph structural information without message passing. The proposed framework is based purely on Multi-Layer Perceptrons (MLPs), where the structural information is only implicitly incorporated as prior knowledge. It first applies structural sparsification to remove potentially uninformative or noisy edges in the neighborhood, and then performs structural self-contrasting in the sparsified neighborhood to learn robust node representations.
  arXiv  Detail & Related papers  (2024-09-09T12:56:02Z)
• A Pure Transformer Pretraining Framework on Text-attributed Graphs [50.833130854272774]
  We introduce a feature-centric pretraining perspective by treating graph structure as a prior. Our framework, Graph Sequence Pretraining with Transformer (GSPT), samples node contexts through random walks. GSPT can be easily adapted to both node classification and link prediction, demonstrating promising empirical success on various datasets.
  arXiv  Detail & Related papers  (2024-06-19T22:30:08Z)
• Hybrid Focal and Full-Range Attention Based Graph Transformers [0.0]
  We present a purely attention-based architecture, namely Focal and Full-Range Graph Transformer (FFGT) FFGT combines the conventional full-range attention with K-hop focal attention on ego-nets to aggregate both global and local information. Our approach enhances the performance of existing Graph Transformers on various open datasets.
  arXiv  Detail & Related papers  (2023-11-08T12:53:07Z)
• Transitivity-Preserving Graph Representation Learning for Bridging Local Connectivity and Role-based Similarity [2.5252594834159643]
  We propose Unified Graph Transformer Networks (UGT) that integrate local and global structural information into fixed-length vector representations. First, UGT learns local structure by identifying the local substructures and aggregating features of the $k$-hop neighborhoods of each node. Third, UGT learns unified representations through self-attention, encoding structural distance and $p$-step transition probability between node pairs.
  arXiv  Detail & Related papers  (2023-08-18T12:49:57Z)
• SGFormer: Simplifying and Empowering Transformers for Large-Graph Representations [75.71298846760303]
  We show that a one-layer attention can bring up surprisingly competitive performance across node property prediction benchmarks. We frame the proposed scheme as Simplified Graph Transformers (SGFormer), which is empowered by a simple attention model. We believe the proposed methodology alone enlightens a new technical path of independent interest for building Transformers on large graphs.
  arXiv  Detail & Related papers  (2023-06-19T08:03:25Z)
• Self-Attention in Colors: Another Take on Encoding Graph Structure in Transformers [25.683127388426175]
  We introduce a novel self-attention mechanism, which we call CSA (Chromatic Self-Attention) We showcase CSA in a fully-attentional graph Transformer CGT (Chromatic Graph Transformer)
  arXiv  Detail & Related papers  (2023-04-21T13:08:53Z)
• Graph Transformer GANs for Graph-Constrained House Generation [223.739067413952]
  We present a novel graph Transformer generative adversarial network (GTGAN) to learn effective graph node relations. The GTGAN learns effective graph node relations in an end-to-end fashion for the challenging graph-constrained house generation task.
  arXiv  Detail & Related papers  (2023-03-14T20:35:45Z)
• Rethinking Graph Transformers with Spectral Attention [13.068288784805901]
  We present the $textitSpectral Attention Network$ (SAN), which uses a learned positional encoding (LPE) to learn the position of each node in a given graph. By leveraging the full spectrum of the Laplacian, our model is theoretically powerful in distinguishing graphs, and can better detect similar sub-structures from their resonance. Our model performs on par or better than state-of-the-art GNNs, and outperforms any attention-based model by a wide margin.
  arXiv  Detail & Related papers  (2021-06-07T18:11:11Z)
• Robust Optimization as Data Augmentation for Large-scale Graphs [117.2376815614148]
  We propose FLAG (Free Large-scale Adversarial Augmentation on Graphs), which iteratively augments node features with gradient-based adversarial perturbations during training. FLAG is a general-purpose approach for graph data, which universally works in node classification, link prediction, and graph classification tasks.
  arXiv  Detail & Related papers  (2020-10-19T21:51:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.