Generalized Graph Prompt: Toward a Unification of Pre-Training and Downstream Tasks on Graphs

• URL: http://arxiv.org/abs/2311.15317v5
• Date: Mon, 26 Aug 2024 10:12:45 GMT
• Title: Generalized Graph Prompt: Toward a Unification of Pre-Training and Downstream Tasks on Graphs
• Authors: Xingtong Yu, Zhenghao Liu, Yuan Fang, Zemin Liu, Sihong Chen, Xinming Zhang,
• Abstract summary: GraphPrompt is a novel pre-training and prompting framework on graphs. It unifies pre-training and downstream tasks into a common task template. It also employs a learnable prompt to assist a downstream task in locating the most relevant knowledge from the pre-trained model.
• Score: 20.406549548630156
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Graph neural networks have emerged as a powerful tool for graph representation learning, but their performance heavily relies on abundant task-specific supervision. To reduce labeling requirement, the "pre-train, prompt" paradigms have become increasingly common. However, existing study of prompting on graphs is limited, lacking a universal treatment to appeal to different downstream tasks. In this paper, we propose GraphPrompt, a novel pre-training and prompting framework on graphs. GraphPrompt not only unifies pre-training and downstream tasks into a common task template but also employs a learnable prompt to assist a downstream task in locating the most relevant knowledge from the pre-trained model in a task-specific manner. To further enhance GraphPrompt in these two stages, we extend it into GraphPrompt+ with two major enhancements. First, we generalize several popular graph pre-training tasks beyond simple link prediction to broaden the compatibility with our task template. Second, we propose a more generalized prompt design that incorporates a series of prompt vectors within every layer of the pre-trained graph encoder, in order to capitalize on the hierarchical information across different layers beyond just the readout layer. Finally, we conduct extensive experiments on five public datasets to evaluate and analyze GraphPrompt and GraphPrompt+.

Related papers

• Instance-Aware Graph Prompt Learning [71.26108600288308]
  We introduce Instance-Aware Graph Prompt Learning (IA-GPL) in this paper. The process involves generating intermediate prompts for each instance using a lightweight architecture. Experiments conducted on multiple datasets and settings showcase the superior performance of IA-GPL compared to state-of-the-art baselines.
  arXiv  Detail & Related papers  (2024-11-26T18:38:38Z)
• MultiGPrompt for Multi-Task Pre-Training and Prompting on Graphs [33.2696184519275]
  MultiGPrompt is a novel multi-task pre-training and prompting framework for graph representation learning. We propose a dual-prompt mechanism consisting of composed and open prompts to leverage task-specific and global pre-training knowledge.
  arXiv  Detail & Related papers  (2023-11-28T02:36:53Z)
• Deep Prompt Tuning for Graph Transformers [55.2480439325792]
  Fine-tuning is resource-intensive and requires storing multiple copies of large models. We propose a novel approach called deep graph prompt tuning as an alternative to fine-tuning. By freezing the pre-trained parameters and only updating the added tokens, our approach reduces the number of free parameters and eliminates the need for multiple model copies.
  arXiv  Detail & Related papers  (2023-09-18T20:12:17Z)
• SimTeG: A Frustratingly Simple Approach Improves Textual Graph Learning [131.04781590452308]
  We present SimTeG, a frustratingly Simple approach for Textual Graph learning. We first perform supervised parameter-efficient fine-tuning (PEFT) on a pre-trained LM on the downstream task. We then generate node embeddings using the last hidden states of finetuned LM.
  arXiv  Detail & Related papers  (2023-08-03T07:00:04Z)
• All in One: Multi-task Prompting for Graph Neural Networks [30.457491401821652]
  We propose a novel multi-task prompting method for graph models. We first unify the format of graph prompts and language prompts with the prompt token, token structure, and inserting pattern. We then study the task space of various graph applications and reformulate downstream problems to the graph-level task.
  arXiv  Detail & Related papers  (2023-07-04T06:27:31Z)
• A Survey of Graph Prompting Methods: Techniques, Applications, and Challenges [25.32529044997131]
  "Pre-train, prompt, predict training" has gained popularity as a way to learn generalizable models with limited labeled data. The design of prompts could be a challenging and time-consuming process in complex tasks. This survey will bridge the gap between graphs and prompt design to facilitate future methodology development.
  arXiv  Detail & Related papers  (2023-03-13T16:49:43Z)
• GraphPrompt: Unifying Pre-Training and Downstream Tasks for Graph Neural Networks [16.455234748896157]
  GraphPrompt is a novel pre-training and prompting framework on graphs. It unifies pre-training and downstream tasks into a common task template. It also employs a learnable prompt to assist a downstream task in locating the most relevant knowledge from the pre-train model.
  arXiv  Detail & Related papers  (2023-02-16T02:51:38Z)
• Graph Pooling for Graph Neural Networks: Progress, Challenges, and Opportunities [128.55790219377315]
  Graph neural networks have emerged as a leading architecture for many graph-level tasks. graph pooling is indispensable for obtaining a holistic graph-level representation of the whole graph.
  arXiv  Detail & Related papers  (2022-04-15T04:02:06Z)
• Neural Graph Matching for Pre-training Graph Neural Networks [72.32801428070749]
  Graph neural networks (GNNs) have been shown powerful capacity at modeling structural data. We present a novel Graph Matching based GNN Pre-Training framework, called GMPT. The proposed method can be applied to fully self-supervised pre-training and coarse-grained supervised pre-training.
  arXiv  Detail & Related papers  (2022-03-03T09:53:53Z)
• GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training [62.73470368851127]
  Graph representation learning has emerged as a powerful technique for addressing real-world problems. We design Graph Contrastive Coding -- a self-supervised graph neural network pre-training framework. We conduct experiments on three graph learning tasks and ten graph datasets.
  arXiv  Detail & Related papers  (2020-06-17T16:18:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.