Can LLMs Alleviate Catastrophic Forgetting in Graph Continual Learning? A Systematic Study

• URL: http://arxiv.org/abs/2505.18697v1
• Date: Sat, 24 May 2025 13:43:29 GMT
• Title: Can LLMs Alleviate Catastrophic Forgetting in Graph Continual Learning? A Systematic Study
• Authors: Ziyang Cheng, Zhixun Li, Yuhan Li, Yixin Song, Kangyi Zhao, Dawei Cheng, Jia Li, Jeffrey Xu Yu,
• Abstract summary: Real-world data, including graph-structure data, often arrives in a streaming manner, which means that learning systems need to continuously acquire new knowledge.<n>We propose a simple-yet-effective method, Simple Graph Continual Learning (SimGCL), that surpasses the previous state-of-the-art GNN-based baseline by around 20%.
• Score: 35.60356938705585
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Nowadays, real-world data, including graph-structure data, often arrives in a streaming manner, which means that learning systems need to continuously acquire new knowledge without forgetting previously learned information. Although substantial existing works attempt to address catastrophic forgetting in graph machine learning, they are all based on training from scratch with streaming data. With the rise of pretrained models, an increasing number of studies have leveraged their strong generalization ability for continual learning. Therefore, in this work, we attempt to answer whether large language models (LLMs) can mitigate catastrophic forgetting in Graph Continual Learning (GCL). We first point out that current experimental setups for GCL have significant flaws, as the evaluation stage may lead to task ID leakage. Then, we evaluate the performance of LLMs in more realistic scenarios and find that even minor modifications can lead to outstanding results. Finally, based on extensive experiments, we propose a simple-yet-effective method, Simple Graph Continual Learning (SimGCL), that surpasses the previous state-of-the-art GNN-based baseline by around 20% under the rehearsal-free constraint. To facilitate reproducibility, we have developed an easy-to-use benchmark LLM4GCL for training and evaluating existing GCL methods. The code is available at: https://github.com/ZhixunLEE/LLM4GCL.

Related papers

• InfoNCE is a Free Lunch for Semantically guided Graph Contrastive Learning [60.61079931266331]
  Graph Contrastive Learning (GCL) continues to play a crucial role in the ongoing surge of research on graph foundation models or LLM as enhancer for graphs.<n>Traditional GCL uses augmentations to define self-supervised tasks, treating augmented pairs as positive samples and others as negative.<n>In this paper, we argue that GCL is essentially a Positive-Unlabeled (PU) learning problem, where the definition of self-supervised tasks should be semantically guided.
  arXiv  Detail & Related papers  (2025-05-07T05:27:36Z)
• A Selective Learning Method for Temporal Graph Continual Learning [18.793135016181804]
  Real-life temporal graphs often introduce new node classes over time, but existing TGL methods assume a fixed set of classes.<n>We define this novel problem as temporal graph continual learning (TGCL), which focuses on efficiently maintaining up-to-date knowledge of old classes.<n>We derive an upper bound on the error caused by such replacement and transform it into objectives for selecting and learning subsets that minimize classification error while preserving the distribution of the full old-class data.
  arXiv  Detail & Related papers  (2025-03-03T14:22:20Z)
• How Do Large Language Models Understand Graph Patterns? A Benchmark for Graph Pattern Comprehension [53.6373473053431]
  This work introduces a benchmark to assess large language models' capabilities in graph pattern tasks.<n>We have developed a benchmark that evaluates whether LLMs can understand graph patterns based on either terminological or topological descriptions.<n>Our benchmark encompasses both synthetic and real datasets, and a variety of models, with a total of 11 tasks and 7 models.
  arXiv  Detail & Related papers  (2024-10-04T04:48:33Z)
• GLBench: A Comprehensive Benchmark for Graph with Large Language Models [41.89444363336435]
  We introduce GLBench, the first comprehensive benchmark for evaluating GraphLLM methods in both supervised and zero-shot scenarios. GLBench provides a fair and thorough evaluation of different categories of GraphLLM methods, along with traditional baselines such as graph neural networks.
  arXiv  Detail & Related papers  (2024-07-10T08:20:47Z)
• Continual Learning on Graphs: Challenges, Solutions, and Opportunities [72.7886669278433]
  We provide a comprehensive review of existing continual graph learning (CGL) algorithms. We compare methods with traditional continual learning techniques and analyze the applicability of the traditional continual learning techniques to forgetting tasks. We will maintain an up-to-date repository featuring a comprehensive list of accessible algorithms.
  arXiv  Detail & Related papers  (2024-02-18T12:24:45Z)
• Benchmarking Sensitivity of Continual Graph Learning for Skeleton-Based Action Recognition [6.14431765787048]
  Continual learning (CL) aims to build machine learning models that can accumulate knowledge continuously over different tasks without retraining from scratch. Previous studies have shown that pre-training graph neural networks (GNN) may lead to negative transfer after fine-tuning. We propose the first continual graph learning benchmark for continual graph learning setting.
  arXiv  Detail & Related papers  (2024-01-31T18:20:42Z)
• Curriculum Learning for Graph Neural Networks: Which Edges Should We Learn First [13.37867275976255]
  We propose a novel strategy to incorporate more edges into training according to their difficulty from easy to hard. We demonstrate the strength of our proposed method in improving the generalization ability and robustness of learned representations.
  arXiv  Detail & Related papers  (2023-10-28T15:35:34Z)
• Exploring the Potential of Large Language Models (LLMs) in Learning on Graphs [59.74814230246034]
  Large Language Models (LLMs) have been proven to possess extensive common knowledge and powerful semantic comprehension abilities. We investigate two possible pipelines: LLMs-as-Enhancers and LLMs-as-Predictors.
  arXiv  Detail & Related papers  (2023-07-07T05:31:31Z)
• Unifying Graph Contrastive Learning with Flexible Contextual Scopes [57.86762576319638]
  We present a self-supervised learning method termed Unifying Graph Contrastive Learning with Flexible Contextual Scopes (UGCL for short) Our algorithm builds flexible contextual representations with contextual scopes by controlling the power of an adjacency matrix. Based on representations from both local and contextual scopes, distL optimises a very simple contrastive loss function for graph representation learning.
  arXiv  Detail & Related papers  (2022-10-17T07:16:17Z)
• Challenging Common Assumptions about Catastrophic Forgetting [13.1202659074346]
  We study the progressive knowledge accumulation (KA) in DNNs trained with gradient-based algorithms in long sequences of tasks with data re-occurrence. We propose a new framework, SCoLe, to investigate KA and discover that catastrophic forgetting has a limited effect on DNNs trained with SGD.
  arXiv  Detail & Related papers  (2022-07-10T21:40:54Z)
• The CLEAR Benchmark: Continual LEArning on Real-World Imagery [77.98377088698984]
  Continual learning (CL) is widely regarded as crucial challenge for lifelong AI. We introduce CLEAR, the first continual image classification benchmark dataset with a natural temporal evolution of visual concepts. We find that a simple unsupervised pre-training step can already boost state-of-the-art CL algorithms.
  arXiv  Detail & Related papers  (2022-01-17T09:09:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.