Empower Text-Attributed Graphs Learning with Large Language Models
(LLMs)
- URL: http://arxiv.org/abs/2310.09872v1
- Date: Sun, 15 Oct 2023 16:04:28 GMT
- Title: Empower Text-Attributed Graphs Learning with Large Language Models
(LLMs)
- Authors: Jianxiang Yu, Yuxiang Ren, Chenghua Gong, Jiaqi Tan, Xiang Li, Xuecang
Zhang
- Abstract summary: We propose a plug-and-play approach to empower text-attributed graphs through node generation using Large Language Models (LLMs)
We employ an edge predictor to capture the structural information inherent in the raw dataset and integrate the newly generated samples into the original graph.
Experiments demonstrate the outstanding performance of our proposed paradigm, particularly in low-shot scenarios.
- Score: 5.920353954082262
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Text-attributed graphs have recently garnered significant attention due to
their wide range of applications in web domains. Existing methodologies employ
word embedding models for acquiring text representations as node features,
which are subsequently fed into Graph Neural Networks (GNNs) for training.
Recently, the advent of Large Language Models (LLMs) has introduced their
powerful capabilities in information retrieval and text generation, which can
greatly enhance the text attributes of graph data. Furthermore, the acquisition
and labeling of extensive datasets are both costly and time-consuming
endeavors. Consequently, few-shot learning has emerged as a crucial problem in
the context of graph learning tasks. In order to tackle this challenge, we
propose a lightweight paradigm called ENG, which adopts a plug-and-play
approach to empower text-attributed graphs through node generation using LLMs.
Specifically, we utilize LLMs to extract semantic information from the labels
and generate samples that belong to these categories as exemplars.
Subsequently, we employ an edge predictor to capture the structural information
inherent in the raw dataset and integrate the newly generated samples into the
original graph. This approach harnesses LLMs for enhancing class-level
information and seamlessly introduces labeled nodes and edges without modifying
the raw dataset, thereby facilitating the node classification task in few-shot
scenarios. Extensive experiments demonstrate the outstanding performance of our
proposed paradigm, particularly in low-shot scenarios. For instance, in the
1-shot setting of the ogbn-arxiv dataset, ENG achieves a 76% improvement over
the baseline model.
Related papers
- All Against Some: Efficient Integration of Large Language Models for Message Passing in Graph Neural Networks [51.19110891434727]
Large Language Models (LLMs) with pretrained knowledge and powerful semantic comprehension abilities have recently shown a remarkable ability to benefit applications using vision and text data.
E-LLaGNN is a framework with an on-demand LLM service that enriches message passing procedure of graph learning by enhancing a limited fraction of nodes from the graph.
arXiv Detail & Related papers (2024-07-20T22:09:42Z) - STAGE: Simplified Text-Attributed Graph Embeddings Using Pre-trained LLMs [1.4624458429745086]
We present a method for enhancing node features in Graph Neural Network (GNN) models that encode Text-Attributed Graphs (TAGs)
Our approach leverages Large-Language Models (LLMs) to generate embeddings for textual attributes.
We show that utilizing pre-trained LLMs as embedding generators provides robust features for ensemble GNN training.
arXiv Detail & Related papers (2024-07-10T08:50:25Z) - A Pure Transformer Pretraining Framework on Text-attributed Graphs [50.833130854272774]
We introduce a feature-centric pretraining perspective by treating graph structure as a prior.
Our framework, Graph Sequence Pretraining with Transformer (GSPT), samples node contexts through random walks.
GSPT can be easily adapted to both node classification and link prediction, demonstrating promising empirical success on various datasets.
arXiv Detail & Related papers (2024-06-19T22:30:08Z) - Unleashing the Potential of Text-attributed Graphs: Automatic Relation Decomposition via Large Language Models [31.443478448031886]
RoSE (Relation-oriented Semantic Edge-decomposition) is a novel framework that decomposes the graph structure by analyzing raw text attributes.
Our framework significantly enhances node classification performance across various datasets, with improvements of up to 16% on the Wisconsin dataset.
arXiv Detail & Related papers (2024-05-28T20:54:47Z) - Parameter-Efficient Tuning Large Language Models for Graph Representation Learning [62.26278815157628]
We introduce Graph-aware.
Efficient Fine-Tuning - GPEFT, a novel approach for efficient graph representation learning.
We use a graph neural network (GNN) to encode structural information from neighboring nodes into a graph prompt.
We validate our approach through comprehensive experiments conducted on 8 different text-rich graphs, observing an average improvement of 2% in hit@1 and Mean Reciprocal Rank (MRR) in link prediction evaluations.
arXiv Detail & Related papers (2024-04-28T18:36:59Z) - Efficient Large Language Models Fine-Tuning On Graphs [23.19795835873144]
Learning from Text-Attributed Graphs (TAGs) has attracted significant attention due to its wide range of real-world applications.
We introduce a novel and efficient approach for the end-to-end fine-tuning of Large Language Models (LLMs) on TAGs, named LEADING.
arXiv Detail & Related papers (2023-12-07T22:35:16Z) - Exploring the Potential of Large Language Models (LLMs) in Learning on
Graphs [59.74814230246034]
Large Language Models (LLMs) have been proven to possess extensive common knowledge and powerful semantic comprehension abilities.
We investigate two possible pipelines: LLMs-as-Enhancers and LLMs-as-Predictors.
arXiv Detail & Related papers (2023-07-07T05:31:31Z) - Harnessing Explanations: LLM-to-LM Interpreter for Enhanced
Text-Attributed Graph Representation Learning [51.90524745663737]
A key innovation is our use of explanations as features, which can be used to boost GNN performance on downstream tasks.
Our method achieves state-of-the-art results on well-established TAG datasets.
Our method significantly speeds up training, achieving a 2.88 times improvement over the closest baseline on ogbn-arxiv.
arXiv Detail & Related papers (2023-05-31T03:18:03Z) - A Robust Stacking Framework for Training Deep Graph Models with
Multifaceted Node Features [61.92791503017341]
Graph Neural Networks (GNNs) with numerical node features and graph structure as inputs have demonstrated superior performance on various supervised learning tasks with graph data.
The best models for such data types in most standard supervised learning settings with IID (non-graph) data are not easily incorporated into a GNN.
Here we propose a robust stacking framework that fuses graph-aware propagation with arbitrary models intended for IID data.
arXiv Detail & Related papers (2022-06-16T22:46:33Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.