LLMs as Zero-shot Graph Learners: Alignment of GNN Representations with LLM Token Embeddings

• URL: http://arxiv.org/abs/2408.14512v2
• Date: Thu, 21 Nov 2024 17:48:25 GMT
• Title: LLMs as Zero-shot Graph Learners: Alignment of GNN Representations with LLM Token Embeddings
• Authors: Duo Wang, Yuan Zuo, Fengzhi Li, Junjie Wu,
• Abstract summary: We introduce a novel framework named Token Embedding-Aligned Graph Language Model (TEA-GLM) We pretrain a GNN, aligning its representations with token embeddings of an LLM. We then train a linear projector that transforms the GNN's representations into a fixed number of graph token embeddings.
• Score: 7.302176015732192
• License:
• Abstract: Zero-shot graph machine learning, especially with graph neural networks (GNNs), has garnered significant interest due to the challenge of scarce labeled data. While methods like self-supervised learning and graph prompt learning have been extensively explored, they often rely on fine-tuning with task-specific labels, limiting their effectiveness in zero-shot scenarios. Inspired by the zero-shot capabilities of instruction-fine-tuned large language models (LLMs), we introduce a novel framework named Token Embedding-Aligned Graph Language Model (TEA-GLM) that leverages LLMs as cross-dataset and cross-task zero-shot learners for graph machine learning. Concretely, we pretrain a GNN, aligning its representations with token embeddings of an LLM. We then train a linear projector that transforms the GNN's representations into a fixed number of graph token embeddings without tuning the LLM. A unified instruction is designed for various graph tasks at different levels, such as node classification (node-level) and link prediction (edge-level). These design choices collectively enhance our method's effectiveness in zero-shot learning, setting it apart from existing methods. Experiments show that our graph token embeddings help the LLM predictor achieve state-of-the-art performance on unseen datasets and tasks compared to other methods using LLMs as predictors.

Related papers

• Enhance Graph Alignment for Large Language Models [33.96082485852042]
  Graph-to-token approaches are popular in enabling Large Language Models to process graph information. Existing methods have a misalignment between self-supervised tasks and supervised downstream tasks. We propose Graph Alignment Large Language Models (GALLM) to benefit from aligned task templates.
  arXiv  Detail & Related papers  (2024-10-15T07:50:34Z)
• How Do Large Language Models Understand Graph Patterns? A Benchmark for Graph Pattern Comprehension [53.6373473053431]
  This work introduces a benchmark to assess large language models' capabilities in graph pattern tasks. We have developed a benchmark that evaluates whether LLMs can understand graph patterns based on either terminological or topological descriptions. Our benchmark encompasses both synthetic and real datasets, and a variety of models, with a total of 11 tasks and 7 models.
  arXiv  Detail & Related papers  (2024-10-04T04:48:33Z)
• All Against Some: Efficient Integration of Large Language Models for Message Passing in Graph Neural Networks [51.19110891434727]
  Large Language Models (LLMs) with pretrained knowledge and powerful semantic comprehension abilities have recently shown a remarkable ability to benefit applications using vision and text data. E-LLaGNN is a framework with an on-demand LLM service that enriches message passing procedure of graph learning by enhancing a limited fraction of nodes from the graph.
  arXiv  Detail & Related papers  (2024-07-20T22:09:42Z)
• Dr.E Bridges Graphs with Large Language Models through Words [12.22063024099311]
  We introduce an end-to-end modality-aligning framework for LLM-graph alignment: Dual-Residual Vector Quantized-Variational AutoEncoder. Our approach is purposefully designed to facilitate token-level alignment with LLMs, enabling an effective translation of the intrinsic '' of graphs into comprehensible natural language.
  arXiv  Detail & Related papers  (2024-06-19T16:43:56Z)
• Can Graph Learning Improve Planning in LLM-based Agents? [61.47027387839096]
  Task planning in language agents is emerging as an important research topic alongside the development of large language models (LLMs) In this paper, we explore graph learning-based methods for task planning, a direction that is to the prevalent focus on prompt design. Our interest in graph learning stems from a theoretical discovery: the biases of attention and auto-regressive loss impede LLMs' ability to effectively navigate decision-making on graphs.
  arXiv  Detail & Related papers  (2024-05-29T14:26:24Z)
• Parameter-Efficient Tuning Large Language Models for Graph Representation Learning [62.26278815157628]
  We introduce Graph-aware. Efficient Fine-Tuning - GPEFT, a novel approach for efficient graph representation learning. We use a graph neural network (GNN) to encode structural information from neighboring nodes into a graph prompt. We validate our approach through comprehensive experiments conducted on 8 different text-rich graphs, observing an average improvement of 2% in hit@1 and Mean Reciprocal Rank (MRR) in link prediction evaluations.
  arXiv  Detail & Related papers  (2024-04-28T18:36:59Z)
• Graph Language Models [18.75364157933661]
  We introduce a novel LM type, the Graph Language Model (GLM), that integrates the strengths of both approaches and mitigates their weaknesses. We design the GLM's architecture to incorporate graph biases, thereby promoting effective knowledge distribution within the graph. Empirical evaluations on relation classification tasks show that GLM embeddings surpass both LM- and GNN-based baselines in supervised and zero-shot setting.
  arXiv  Detail & Related papers  (2024-01-13T16:09:49Z)
• Large Language Models as Topological Structure Enhancers for Text-Attributed Graphs [4.487720716313697]
  Large language models (LLMs) have revolutionized the field of natural language processing (NLP) This work explores how to leverage the information retrieval and text generation capabilities of LLMs to refine/enhance the topological structure of text-attributed graphs (TAGs) under the node classification setting.
  arXiv  Detail & Related papers  (2023-11-24T07:53:48Z)
• Integrating Graphs with Large Language Models: Methods and Prospects [68.37584693537555]
  Large language models (LLMs) have emerged as frontrunners, showcasing unparalleled prowess in diverse applications. Merging the capabilities of LLMs with graph-structured data has been a topic of keen interest. This paper bifurcates such integrations into two predominant categories.
  arXiv  Detail & Related papers  (2023-10-09T07:59:34Z)
• Exploring the Potential of Large Language Models (LLMs) in Learning on Graphs [59.74814230246034]
  Large Language Models (LLMs) have been proven to possess extensive common knowledge and powerful semantic comprehension abilities. We investigate two possible pipelines: LLMs-as-Enhancers and LLMs-as-Predictors.
  arXiv  Detail & Related papers  (2023-07-07T05:31:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.