LPNL: Scalable Link Prediction with Large Language Models

• URL: http://arxiv.org/abs/2401.13227v3
• Date: Tue, 20 Feb 2024 03:53:06 GMT
• Title: LPNL: Scalable Link Prediction with Large Language Models
• Authors: Baolong Bi, Shenghua Liu, Yiwei Wang, Lingrui Mei and Xueqi Cheng
• Abstract summary: This work focuses on the link prediction task and introduces $textbfLPNL$ (Link Prediction via Natural Language), a framework based on large language models. We design novel prompts for link prediction that articulate graph details in natural language. We propose a two-stage sampling pipeline to extract crucial information from the graphs, and a divide-and-conquer strategy to control the input tokens.
• Score: 46.65436204783482
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Exploring the application of large language models (LLMs) to graph learning is a emerging endeavor. However, the vast amount of information inherent in large graphs poses significant challenges to this process. This work focuses on the link prediction task and introduces $\textbf{LPNL}$ (Link Prediction via Natural Language), a framework based on large language models designed for scalable link prediction on large-scale heterogeneous graphs. We design novel prompts for link prediction that articulate graph details in natural language. We propose a two-stage sampling pipeline to extract crucial information from the graphs, and a divide-and-conquer strategy to control the input tokens within predefined limits, addressing the challenge of overwhelming information. We fine-tune a T5 model based on our self-supervised learning designed for link prediction. Extensive experimental results demonstrate that LPNL outperforms multiple advanced baselines in link prediction tasks on large-scale graphs.

Related papers

• Parameter-Efficient Tuning Large Language Models for Graph Representation Learning [62.26278815157628]
  We introduce Graph-aware. Efficient Fine-Tuning - GPEFT, a novel approach for efficient graph representation learning. We use a graph neural network (GNN) to encode structural information from neighboring nodes into a graph prompt. We validate our approach through comprehensive experiments conducted on 8 different text-rich graphs, observing an average improvement of 2% in hit@1 and Mean Reciprocal Rank (MRR) in link prediction evaluations.
  arXiv  Detail & Related papers  (2024-04-28T18:36:59Z)
• CORE: Data Augmentation for Link Prediction via Information Bottleneck [25.044734252779975]
  Link prediction (LP) is a fundamental task in graph representation learning. We propose a novel data augmentation method, COmplete and REduce (CORE) to learn compact and predictive augmentations for LP models.
  arXiv  Detail & Related papers  (2024-04-17T03:20:42Z)
• Narrating Causal Graphs with Large Language Models [1.437446768735628]
  This work explores the capability of large pretrained language models to generate text from causal graphs. The causal reasoning encoded in these graphs can support applications as diverse as healthcare or marketing. Results suggest users of generative AI can deploy future applications faster since similar performances are obtained when training a model with only a few examples.
  arXiv  Detail & Related papers  (2024-03-11T19:19:59Z)
• Graph-Aware Language Model Pre-Training on a Large Graph Corpus Can Help Multiple Graph Applications [38.83545631999851]
  We propose a framework of graph-aware language model pre-training on a large graph corpus. We conduct experiments on Amazon's real internal datasets and large public datasets.
  arXiv  Detail & Related papers  (2023-06-05T04:46:44Z)
• Learning Large Graph Property Prediction via Graph Segment Training [61.344814074335304]
  We propose a general framework that allows learning large graph property prediction with a constant memory footprint. We refine the GST paradigm by introducing a historical embedding table to efficiently obtain embeddings for segments not sampled for backpropagation. Our experiments show that GST-EFD is both memory-efficient and fast, while offering a slight boost on test accuracy over a typical full graph training regime.
  arXiv  Detail & Related papers  (2023-05-21T02:53:25Z)
• Neural Graph Matching for Pre-training Graph Neural Networks [72.32801428070749]
  Graph neural networks (GNNs) have been shown powerful capacity at modeling structural data. We present a novel Graph Matching based GNN Pre-Training framework, called GMPT. The proposed method can be applied to fully self-supervised pre-training and coarse-grained supervised pre-training.
  arXiv  Detail & Related papers  (2022-03-03T09:53:53Z)
• Node Feature Extraction by Self-Supervised Multi-scale Neighborhood Prediction [123.20238648121445]
  We propose a new self-supervised learning framework, Graph Information Aided Node feature exTraction (GIANT) GIANT makes use of the eXtreme Multi-label Classification (XMC) formalism, which is crucial for fine-tuning the language model based on graph information. We demonstrate the superior performance of GIANT over the standard GNN pipeline on Open Graph Benchmark datasets.
  arXiv  Detail & Related papers  (2021-10-29T19:55:12Z)
• Ensembling Graph Predictions for AMR Parsing [28.625065956013778]
  In many machine learning tasks, models are trained to predict structure data such as graphs. In this work, we formalize this problem as mining the largest graph that is the most supported by a collection of graph predictions. We show that the proposed approach can combine the strength of state-of-the-art AMRs to create new predictions that are more accurate than any individual models in five standard benchmark datasets.
  arXiv  Detail & Related papers  (2021-10-18T09:35:39Z)
• How Neural Processes Improve Graph Link Prediction [35.652234989200956]
  We propose a meta-learning approach with graph neural networks for link prediction: Neural Processes for Graph Neural Networks (NPGNN) NPGNN can perform both transductive and inductive learning tasks and adapt to patterns in a large new graph after training with a small subgraph.
  arXiv  Detail & Related papers  (2021-09-30T07:35:13Z)
• Learning to Extrapolate Knowledge: Transductive Few-shot Out-of-Graph Link Prediction [69.1473775184952]
  We introduce a realistic problem of few-shot out-of-graph link prediction. We tackle this problem with a novel transductive meta-learning framework. We validate our model on multiple benchmark datasets for knowledge graph completion and drug-drug interaction prediction.
  arXiv  Detail & Related papers  (2020-06-11T17:42:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.