PRODIGY: Enabling In-context Learning Over Graphs
- URL: http://arxiv.org/abs/2305.12600v1
- Date: Sun, 21 May 2023 23:16:30 GMT
- Title: PRODIGY: Enabling In-context Learning Over Graphs
- Authors: Qian Huang, Hongyu Ren, Peng Chen, Gregor Kr\v{z}manc, Daniel Zeng,
Percy Liang, Jure Leskovec
- Abstract summary: In-context learning is the ability of a pretrained model to adapt to novel and diverse downstream tasks.
We develop PRODIGY, the first pretraining framework that enables in-context learning over graphs.
- Score: 112.19056551153454
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: In-context learning is the ability of a pretrained model to adapt to novel
and diverse downstream tasks by conditioning on prompt examples, without
optimizing any parameters. While large language models have demonstrated this
ability, how in-context learning could be performed over graphs is unexplored.
In this paper, we develop \textbf{Pr}etraining \textbf{O}ver \textbf{D}iverse
\textbf{I}n-Context \textbf{G}raph S\textbf{y}stems (PRODIGY), the first
pretraining framework that enables in-context learning over graphs. The key
idea of our framework is to formulate in-context learning over graphs with a
novel \emph{prompt graph} representation, which connects prompt examples and
queries. We then propose a graph neural network architecture over the prompt
graph and a corresponding family of in-context pretraining objectives. With
PRODIGY, the pretrained model can directly perform novel downstream
classification tasks on unseen graphs via in-context learning. We provide
empirical evidence of the effectiveness of our framework by showcasing its
strong in-context learning performance on tasks involving citation networks and
knowledge graphs. Our approach outperforms the in-context learning accuracy of
contrastive pretraining baselines with hard-coded adaptation by 18\% on average
across all setups. Moreover, it also outperforms standard finetuning with
limited data by 33\% on average with in-context learning.
Related papers
- Instance-Aware Graph Prompt Learning [71.26108600288308]
We introduce Instance-Aware Graph Prompt Learning (IA-GPL) in this paper.
The process involves generating intermediate prompts for each instance using a lightweight architecture.
Experiments conducted on multiple datasets and settings showcase the superior performance of IA-GPL compared to state-of-the-art baselines.
arXiv Detail & Related papers (2024-11-26T18:38:38Z) - Node Level Graph Autoencoder: Unified Pretraining for Textual Graph Learning [45.70767623846523]
We propose a novel unified unsupervised learning autoencoder framework, named Node Level Graph AutoEncoder (NodeGAE)
We employ language models as the backbone of the autoencoder, with pretraining on text reconstruction.
Our method maintains simplicity in the training process and demonstrates generalizability across diverse textual graphs and downstream tasks.
arXiv Detail & Related papers (2024-08-09T14:57:53Z) - GRENADE: Graph-Centric Language Model for Self-Supervised Representation
Learning on Text-Attributed Graphs [22.282756544376493]
We develop a novel Graph-Centric Language model, GRENADE, to solve the problem of self-supervised representation learning on text-attributed graphs.
GRENADE exploits the synergistic effect of both pre-trained language model and graph neural network by optimizing with two specialized self-supervised learning algorithms.
The proposed graph-centric self-supervised learning algorithms effectively help GRENADE to capture informative textual semantics as well as structural context information on text-attributed graphs.
arXiv Detail & Related papers (2023-10-23T17:18:35Z) - SimTeG: A Frustratingly Simple Approach Improves Textual Graph Learning [131.04781590452308]
We present SimTeG, a frustratingly Simple approach for Textual Graph learning.
We first perform supervised parameter-efficient fine-tuning (PEFT) on a pre-trained LM on the downstream task.
We then generate node embeddings using the last hidden states of finetuned LM.
arXiv Detail & Related papers (2023-08-03T07:00:04Z) - Enhancing Dialogue Generation via Dynamic Graph Knowledge Aggregation [23.54754465832362]
In conventional graph neural networks (GNNs) message passing on a graph is independent from text.
This training regime leads to a semantic gap between graph knowledge and text.
We propose a novel framework for knowledge graph enhanced dialogue generation.
arXiv Detail & Related papers (2023-06-28T13:21:00Z) - ConGraT: Self-Supervised Contrastive Pretraining for Joint Graph and Text Embeddings [20.25180279903009]
We propose Contrastive Graph-Text pretraining (ConGraT) for jointly learning separate representations of texts and nodes in a text-attributed graph (TAG)
Our method trains a language model (LM) and a graph neural network (GNN) to align their representations in a common latent space using a batch-wise contrastive learning objective inspired by CLIP.
Experiments demonstrate that ConGraT outperforms baselines on various downstream tasks, including node and text category classification, link prediction, and language modeling.
arXiv Detail & Related papers (2023-05-23T17:53:30Z) - Neural Graph Matching for Pre-training Graph Neural Networks [72.32801428070749]
Graph neural networks (GNNs) have been shown powerful capacity at modeling structural data.
We present a novel Graph Matching based GNN Pre-Training framework, called GMPT.
The proposed method can be applied to fully self-supervised pre-training and coarse-grained supervised pre-training.
arXiv Detail & Related papers (2022-03-03T09:53:53Z) - GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training [62.73470368851127]
Graph representation learning has emerged as a powerful technique for addressing real-world problems.
We design Graph Contrastive Coding -- a self-supervised graph neural network pre-training framework.
We conduct experiments on three graph learning tasks and ten graph datasets.
arXiv Detail & Related papers (2020-06-17T16:18:35Z) - Exploiting Structured Knowledge in Text via Graph-Guided Representation
Learning [73.0598186896953]
We present two self-supervised tasks learning over raw text with the guidance from knowledge graphs.
Building upon entity-level masked language models, our first contribution is an entity masking scheme.
In contrast to existing paradigms, our approach uses knowledge graphs implicitly, only during pre-training.
arXiv Detail & Related papers (2020-04-29T14:22:42Z) - A Graph Convolutional Topic Model for Short and Noisy Text Streams [2.578242050187029]
We propose a novel graph convolutional topic model (GCTM) which integrates graph convolutional networks (GCN) into a topic model.
We conduct extensive experiments to evaluate our method with both a human knowledge graph (Wordnet) and a graph built from pre-trained word embeddings (Word2vec)
Our method achieves significantly better performances than state-of-the-art baselines in terms of probabilistic predictive measure and topic coherence.
arXiv Detail & Related papers (2020-03-13T05:09:00Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.