Edge-augmented Graph Transformers: Global Self-attention is Enough for
Graphs
- URL: http://arxiv.org/abs/2108.03348v1
- Date: Sat, 7 Aug 2021 02:18:11 GMT
- Title: Edge-augmented Graph Transformers: Global Self-attention is Enough for
Graphs
- Authors: Md Shamim Hussain, Mohammed J. Zaki and Dharmashankar Subramanian
- Abstract summary: We propose a simple yet powerful extension to the transformer - residual edge channels.
The resultant framework, which we call Edge-augmented Graph Transformer (EGT), can directly accept, process and output structural information as well as node information.
Our framework, which relies on global node feature aggregation, achieves better performance compared to Graph Convolutional Networks (GCN)
- Score: 24.796242917673755
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: Transformer neural networks have achieved state-of-the-art results for
unstructured data such as text and images but their adoption for
graph-structured data has been limited. This is partly due to the difficulty in
incorporating complex structural information in the basic transformer
framework. We propose a simple yet powerful extension to the transformer -
residual edge channels. The resultant framework, which we call Edge-augmented
Graph Transformer (EGT), can directly accept, process and output structural
information as well as node information. This simple addition allows us to use
global self-attention, the key element of transformers, directly for graphs and
comes with the benefit of long-range interaction among nodes. Moreover, the
edge channels allow the structural information to evolve from layer to layer,
and prediction tasks on edges can be derived directly from these channels. In
addition to that, we introduce positional encodings based on Singular Value
Decomposition which can improve the performance of EGT. Our framework, which
relies on global node feature aggregation, achieves better performance compared
to Graph Convolutional Networks (GCN), which rely on local feature aggregation
within a neighborhood. We verify the performance of EGT in a supervised
learning setting on a wide range of experiments on benchmark datasets. Our
findings indicate that convolutional aggregation is not an essential inductive
bias for graphs and global self-attention can serve as a flexible and adaptive
alternative to graph convolution.
Related papers
- A Pure Transformer Pretraining Framework on Text-attributed Graphs [50.833130854272774]
We introduce a feature-centric pretraining perspective by treating graph structure as a prior.
Our framework, Graph Sequence Pretraining with Transformer (GSPT), samples node contexts through random walks.
GSPT can be easily adapted to both node classification and link prediction, demonstrating promising empirical success on various datasets.
arXiv Detail & Related papers (2024-06-19T22:30:08Z) - Hybrid Focal and Full-Range Attention Based Graph Transformers [0.0]
We present a purely attention-based architecture, namely Focal and Full-Range Graph Transformer (FFGT)
FFGT combines the conventional full-range attention with K-hop focal attention on ego-nets to aggregate both global and local information.
Our approach enhances the performance of existing Graph Transformers on various open datasets.
arXiv Detail & Related papers (2023-11-08T12:53:07Z) - Network Alignment with Transferable Graph Autoencoders [79.89704126746204]
We propose a novel graph autoencoder architecture designed to extract powerful and robust node embeddings.
We prove that the generated embeddings are associated with the eigenvalues and eigenvectors of the graphs.
Our proposed framework also leverages transfer learning and data augmentation to achieve efficient network alignment at a very large scale without retraining.
arXiv Detail & Related papers (2023-10-05T02:58:29Z) - Transitivity-Preserving Graph Representation Learning for Bridging Local
Connectivity and Role-based Similarity [2.5252594834159643]
We propose Unified Graph Transformer Networks (UGT) that integrate local and global structural information into fixed-length vector representations.
First, UGT learns local structure by identifying the local substructures and aggregating features of the $k$-hop neighborhoods of each node.
Third, UGT learns unified representations through self-attention, encoding structural distance and $p$-step transition probability between node pairs.
arXiv Detail & Related papers (2023-08-18T12:49:57Z) - SGFormer: Simplifying and Empowering Transformers for Large-Graph
Representations [78.97396248946174]
We show that a one-layer attention can bring up surprisingly competitive performance across node property prediction benchmarks.
We frame the proposed scheme as Simplified Graph Transformers (SGFormer), which is empowered by a simple attention model.
We believe the proposed methodology alone enlightens a new technical path of independent interest for building Transformers on large graphs.
arXiv Detail & Related papers (2023-06-19T08:03:25Z) - Self-Attention in Colors: Another Take on Encoding Graph Structure in
Transformers [25.683127388426175]
We introduce a novel self-attention mechanism, which we call CSA (Chromatic Self-Attention)
We showcase CSA in a fully-attentional graph Transformer CGT (Chromatic Graph Transformer)
arXiv Detail & Related papers (2023-04-21T13:08:53Z) - Graph Transformer GANs for Graph-Constrained House Generation [223.739067413952]
We present a novel graph Transformer generative adversarial network (GTGAN) to learn effective graph node relations.
The GTGAN learns effective graph node relations in an end-to-end fashion for the challenging graph-constrained house generation task.
arXiv Detail & Related papers (2023-03-14T20:35:45Z) - Spectral Graph Convolutional Networks With Lifting-based Adaptive Graph
Wavelets [81.63035727821145]
Spectral graph convolutional networks (SGCNs) have been attracting increasing attention in graph representation learning.
We propose a novel class of spectral graph convolutional networks that implement graph convolutions with adaptive graph wavelets.
arXiv Detail & Related papers (2021-08-03T17:57:53Z) - Rethinking Graph Transformers with Spectral Attention [13.068288784805901]
We present the $textitSpectral Attention Network$ (SAN), which uses a learned positional encoding (LPE) to learn the position of each node in a given graph.
By leveraging the full spectrum of the Laplacian, our model is theoretically powerful in distinguishing graphs, and can better detect similar sub-structures from their resonance.
Our model performs on par or better than state-of-the-art GNNs, and outperforms any attention-based model by a wide margin.
arXiv Detail & Related papers (2021-06-07T18:11:11Z) - Robust Optimization as Data Augmentation for Large-scale Graphs [117.2376815614148]
We propose FLAG (Free Large-scale Adversarial Augmentation on Graphs), which iteratively augments node features with gradient-based adversarial perturbations during training.
FLAG is a general-purpose approach for graph data, which universally works in node classification, link prediction, and graph classification tasks.
arXiv Detail & Related papers (2020-10-19T21:51:47Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.