GraphLAMA: Enabling Efficient Adaptation of Graph Language Models with Limited Annotations

• URL: http://arxiv.org/abs/2506.21559v1
• Date: Wed, 11 Jun 2025 16:38:01 GMT
• Title: GraphLAMA: Enabling Efficient Adaptation of Graph Language Models with Limited Annotations
• Authors: Junze Chen, Cheng Yang, Shujie Li, Zhiqiang Zhang, Yawen Li, Junping Du, Chuan Shi,
• Abstract summary: Large language models (LLMs) have demonstrated their strong capabilities in various domains, and have been recently integrated for graph analysis as graph language models (GLMs)<n>With LLMs as the predictor, some GLMs can interpret unseen tasks described by natural language, and learn from a few examples in the prompts without parameter tuning, known as in-context learning (ICL)<n>We propose GraphLAMA method, with its model backbone and learning schemes specialized for efficient tuning and inference.
• Score: 46.15515676751084
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) have demonstrated their strong capabilities in various domains, and have been recently integrated for graph analysis as graph language models (GLMs). With LLMs as the predictor, some GLMs can interpret unseen tasks described by natural language, and learn from a few examples in the prompts without parameter tuning, known as in-context learning (ICL). Another subset of GLMs utilizes abundant training labels to enhance model performance, known as instruction tuning. However, we argue that ICL on graphs has effectiveness issues due to fixed parameters and efficiency issues due to long context. Meanwhile, the large amount of labeled data required for instruction tuning can be difficult to obtain in real-world scenarios. To this end, we aim to introduce an extra parameter adaptation stage that can efficiently tailor GLMs to an unseen graph and task with only a few labeled examples, in exchange for better prediction accuracy and faster inference speed. For implementation, in this paper we propose GraphLAMA method, with its model backbone and learning schemes specialized for efficient tuning and inference. Specifically, for model backbone, we use a graph neural network (GNN) with several well-designed components to transform nodes into the representation space of LLM tokens. Task instructions can then be represented as a mixture of node and language tokens. In the pre-training stage, model parameters except the LLM will be trained with different tasks to capture general knowledge. In the adaptation stage, only a few pre-trained parameters will be updated based on few-shot examples. Extensive experiments on few/zero-shot node classification and summary generation show that our proposed GraphLAMA achieves state-of-the-art performance with 4.91% absolution improvement in accuracy. Compared with ICL, our inference speed can be 10 times faster under 5-shot setting.

Related papers

• Adaptive Tokenization: On the Hop-Overpriority Problem in Tokenized Graph Learning Models [59.98982735197465]
  Tokenized Graph Learning Models (TGLMs) convert graphs into ordered token lists for scalable processing.<n>TGLMs rely on hand-designed token lists and their adaptability to diverse graph learning scenarios remains unexplored.<n>We propose Learnable Graph Token List (LGTL), a plug-and-play module to replace hand-designed token lists in TGLMs.
  arXiv  Detail & Related papers  (2025-05-19T06:25:33Z)
• Scalability Matters: Overcoming Challenges in InstructGLM with Similarity-Degree-Based Sampling [1.2805157669888096]
  We propose SDM-InstructGLM, a novel instruction-tuned Graph Language Model (InstructGLM) framework that enhances scalability and efficiency without relying on GNNs.<n>Our method introduces a similarity-degree-based biased random walk mechanism, which selectively samples and encodes graph information based on node-feature similarity and degree centrality.<n>Our results demonstrate the feasibility of LLM-only graph processing, enabling scalable and interpretable Graph Language Models (GLMs) optimized through instruction-based fine-tuning.
  arXiv  Detail & Related papers  (2025-05-02T06:08:21Z)
• Training Large Recommendation Models via Graph-Language Token Alignment [53.3142545812349]
  We propose a novel framework to train Large Recommendation models via Graph-Language Token Alignment.<n>By aligning item and user nodes from the interaction graph with pretrained LLM tokens, GLTA effectively leverages the reasoning abilities of LLMs.<n> Furthermore, we introduce Graph-Language Logits Matching (GLLM) to optimize token alignment for end-to-end item prediction.
  arXiv  Detail & Related papers  (2025-02-26T02:19:10Z)
• Can Graph Neural Networks Learn Language with Extremely Weak Text Supervision? [62.12375949429938]
  We propose a multi-modal prompt learning paradigm to adapt pre-trained Graph Neural Networks to downstream tasks and data.<n>Our new paradigm embeds the graphs directly in the same space as the Large Language Models (LLMs) by learning both graph prompts and text prompts simultaneously.<n>We build the first CLIP-style zero-shot classification prototype that can generalize GNNs to unseen classes with extremely weak text supervision.
  arXiv  Detail & Related papers  (2024-12-11T08:03:35Z)
• Enhance Graph Alignment for Large Language Models [33.96082485852042]
  Graph-to-token approaches are popular in enabling Large Language Models to process graph information. Existing methods have a misalignment between self-supervised tasks and supervised downstream tasks. We propose Graph Alignment Large Language Models (GALLM) to benefit from aligned task templates.
  arXiv  Detail & Related papers  (2024-10-15T07:50:34Z)
• Can Large Language Models Analyze Graphs like Professionals? A Benchmark, Datasets and Models [88.4320775961431]
  We introduce ProGraph, a benchmark for large language models (LLMs) to process graphs.<n>Our findings reveal that the performance of current LLMs is unsatisfactory, with the best model achieving only 36% accuracy.<n>We propose LLM4Graph datasets, which include crawled documents and auto-generated codes based on 6 widely used graph libraries.
  arXiv  Detail & Related papers  (2024-09-29T11:38:45Z)
• LLMs as Zero-shot Graph Learners: Alignment of GNN Representations with LLM Token Embeddings [7.302176015732192]
  We introduce a novel framework named Token Embedding-Aligned Graph Language Model (TEA-GLM)<n>We pretrain a GNN, aligning its representations with token embeddings of an LLM.<n>We then train a linear projector that transforms the GNN's representations into a fixed number of graph token embeddings.
  arXiv  Detail & Related papers  (2024-08-25T04:32:45Z)
• All Against Some: Efficient Integration of Large Language Models for Message Passing in Graph Neural Networks [51.19110891434727]
  Large Language Models (LLMs) with pretrained knowledge and powerful semantic comprehension abilities have recently shown a remarkable ability to benefit applications using vision and text data. E-LLaGNN is a framework with an on-demand LLM service that enriches message passing procedure of graph learning by enhancing a limited fraction of nodes from the graph.
  arXiv  Detail & Related papers  (2024-07-20T22:09:42Z)
• Parameter-Efficient Tuning Large Language Models for Graph Representation Learning [62.26278815157628]
  We introduce Graph-aware. Efficient Fine-Tuning - GPEFT, a novel approach for efficient graph representation learning. We use a graph neural network (GNN) to encode structural information from neighboring nodes into a graph prompt. We validate our approach through comprehensive experiments conducted on 8 different text-rich graphs, observing an average improvement of 2% in hit@1 and Mean Reciprocal Rank (MRR) in link prediction evaluations.
  arXiv  Detail & Related papers  (2024-04-28T18:36:59Z)
• Can GNN be Good Adapter for LLMs? [7.18511200494162]
  Text-attributed graphs (TAGs) have broad applications in social media, recommendation systems, etc. We propose GraphAdapter, which uses a graph neural network (GNN) as an efficient adapter in collaboration with large language models (LLMs) to tackle TAGs. Through extensive experiments across multiple real-world TAGs, GraphAdapter based on Llama 2 gains an average improvement of approximately 5% in terms of node classification.
  arXiv  Detail & Related papers  (2024-02-20T13:13:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.