GraphGPT: Graph Learning with Generative Pre-trained Transformers
- URL: http://arxiv.org/abs/2401.00529v1
- Date: Sun, 31 Dec 2023 16:19:30 GMT
- Title: GraphGPT: Graph Learning with Generative Pre-trained Transformers
- Authors: Qifang Zhao, Weidong Ren, Tianyu Li, Xiaoxiao Xu and Hong Liu
- Abstract summary: We introduce textitGraphGPT, a novel model for Graph learning by self-supervised Generative Pre-training Transformers.
Our model transforms each graph or sampled subgraph into a sequence of tokens representing the node, edge and attributes.
The generative pre-training enables us to train GraphGPT up to 400M+ parameters with consistently increasing performance.
- Score: 9.862004020075126
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: We introduce \textit{GraphGPT}, a novel model for Graph learning by
self-supervised Generative Pre-training Transformers. Our model transforms each
graph or sampled subgraph into a sequence of tokens representing the node, edge
and attributes reversibly using the Eulerian path first. Then we feed the
tokens into a standard transformer decoder and pre-train it with the
next-token-prediction (NTP) task. Lastly, we fine-tune the GraphGPT model with
the supervised tasks. This intuitive, yet effective model achieves superior or
close results to the state-of-the-art methods for the graph-, edge- and
node-level tasks on the large scale molecular dataset PCQM4Mv2, the
protein-protein association dataset ogbl-ppa and the ogbn-proteins dataset from
the Open Graph Benchmark (OGB). Furthermore, the generative pre-training
enables us to train GraphGPT up to 400M+ parameters with consistently
increasing performance, which is beyond the capability of GNNs and previous
graph transformers. The source code and pre-trained checkpoints will be
released soon\footnote{\url{https://github.com/alibaba/graph-gpt}} to pave the
way for the graph foundation model research, and also to assist the scientific
discovery in pharmaceutical, chemistry, material and bio-informatics domains,
etc.
Related papers
- Learning Graph Quantized Tokenizers for Transformers [28.79505338383552]
Graph Transformers (GTs) have emerged as a leading model in deep learning, outperforming Graph Neural Networks (GNNs) in various graph learning tasks.
We introduce GQT (textbfGraph textbfQuantized textbfTokenizer), which decouples tokenizer training from Transformer training by leveraging graph self-supervised learning.
By combining the GQT with token modulation, a Transformer encoder achieves state-of-the-art performance on 16 out of 18 benchmarks, including large-scale homophilic and heterophilic datasets.
arXiv Detail & Related papers (2024-10-17T17:38:24Z) - Deep Prompt Tuning for Graph Transformers [55.2480439325792]
Fine-tuning is resource-intensive and requires storing multiple copies of large models.
We propose a novel approach called deep graph prompt tuning as an alternative to fine-tuning.
By freezing the pre-trained parameters and only updating the added tokens, our approach reduces the number of free parameters and eliminates the need for multiple model copies.
arXiv Detail & Related papers (2023-09-18T20:12:17Z) - SimTeG: A Frustratingly Simple Approach Improves Textual Graph Learning [131.04781590452308]
We present SimTeG, a frustratingly Simple approach for Textual Graph learning.
We first perform supervised parameter-efficient fine-tuning (PEFT) on a pre-trained LM on the downstream task.
We then generate node embeddings using the last hidden states of finetuned LM.
arXiv Detail & Related papers (2023-08-03T07:00:04Z) - PatchGT: Transformer over Non-trainable Clusters for Learning Graph
Representations [18.203910156450085]
We propose a new Transformer-based graph neural network: Patch Graph Transformer (PatchGT)
Unlike previous transformer-based models for learning graph representations, PatchGT learns from non-trainable graph patches, not from nodes directly.
PatchGT achieves higher than 1-WL-type GNNs, and the empirical study shows that PatchGT achieves competitive performances on benchmark datasets.
arXiv Detail & Related papers (2022-11-26T01:17:23Z) - Self-supervised Graph Masking Pre-training for Graph-to-Text Generation [5.108327983929205]
Large-scale pre-trained language models (PLMs) have advanced Graph-to-Text (G2T) generation.
We propose graph masking pre-training strategies that neither require supervision signals nor adjust the architecture of the underlying pre-trained encoder-decoder model.
Our approach achieves new state-of-the-art results on WebNLG+ 2020 and EventNarrative G2T generation datasets.
arXiv Detail & Related papers (2022-10-19T14:44:56Z) - Pure Transformers are Powerful Graph Learners [51.36884247453605]
We show that standard Transformers without graph-specific modifications can lead to promising results in graph learning both in theory and practice.
We prove that this approach is theoretically at least as expressive as an invariant graph network (2-IGN) composed of equivariant linear layers.
Our method coined Tokenized Graph Transformer (TokenGT) achieves significantly better results compared to GNN baselines and competitive results.
arXiv Detail & Related papers (2022-07-06T08:13:06Z) - Neural Graph Matching for Pre-training Graph Neural Networks [72.32801428070749]
Graph neural networks (GNNs) have been shown powerful capacity at modeling structural data.
We present a novel Graph Matching based GNN Pre-Training framework, called GMPT.
The proposed method can be applied to fully self-supervised pre-training and coarse-grained supervised pre-training.
arXiv Detail & Related papers (2022-03-03T09:53:53Z) - GPT-GNN: Generative Pre-Training of Graph Neural Networks [93.35945182085948]
Graph neural networks (GNNs) have been demonstrated to be powerful in modeling graph-structured data.
We present the GPT-GNN framework to initialize GNNs by generative pre-training.
We show that GPT-GNN significantly outperforms state-of-the-art GNN models without pre-training by up to 9.1% across various downstream tasks.
arXiv Detail & Related papers (2020-06-27T20:12:33Z) - Heuristic Semi-Supervised Learning for Graph Generation Inspired by
Electoral College [80.67842220664231]
We propose a novel pre-processing technique, namely ELectoral COllege (ELCO), which automatically expands new nodes and edges to refine the label similarity within a dense subgraph.
In all setups tested, our method boosts the average score of base models by a large margin of 4.7 points, as well as consistently outperforms the state-of-the-art.
arXiv Detail & Related papers (2020-06-10T14:48:48Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.