Overcoming Catastrophic Forgetting in Graph Neural Networks with Experience Replay

• URL: http://arxiv.org/abs/2003.09908v2
• Date: Mon, 8 Feb 2021 08:26:58 GMT
• Title: Overcoming Catastrophic Forgetting in Graph Neural Networks with Experience Replay
• Authors: Fan Zhou and Chengtai Cao
• Abstract summary: Graph Neural Networks (GNNs) have recently received significant research attention due to their superior performance on a variety of graph-related learning tasks. In this work, we investigate the question can GNNs be applied to continuously learning tasks?
• Score: 16.913443823792022
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Graph Neural Networks (GNNs) have recently received significant research attention due to their superior performance on a variety of graph-related learning tasks. Most of the current works focus on either static or dynamic graph settings, addressing a single particular task, e.g., node/graph classification, link prediction. In this work, we investigate the question: can GNNs be applied to continuously learning a sequence of tasks? Towards that, we explore the Continual Graph Learning (CGL) paradigm and present the Experience Replay based framework ER-GNN for CGL to alleviate the catastrophic forgetting problem in existing GNNs. ER-GNN stores knowledge from previous tasks as experiences and replays them when learning new tasks to mitigate the catastrophic forgetting issue. We propose three experience node selection strategies: mean of feature, coverage maximization, and influence maximization, to guide the process of selecting experience nodes. Extensive experiments on three benchmark datasets demonstrate the effectiveness of our ER-GNN and shed light on the incremental graph (non-Euclidean) structure learning.

Related papers

• Information Flow in Graph Neural Networks: A Clinical Triage Use Case [49.86931948849343]
  Graph Neural Networks (GNNs) have gained popularity in healthcare and other domains due to their ability to process multi-modal and multi-relational graphs. We investigate how the flow of embedding information within GNNs affects the prediction of links in Knowledge Graphs (KGs) Our results demonstrate that incorporating domain knowledge into the GNN connectivity leads to better performance than using the same connectivity as the KG or allowing unconstrained embedding propagation.
  arXiv  Detail & Related papers  (2023-09-12T09:18:12Z)
• SimTeG: A Frustratingly Simple Approach Improves Textual Graph Learning [131.04781590452308]
  We present SimTeG, a frustratingly Simple approach for Textual Graph learning. We first perform supervised parameter-efficient fine-tuning (PEFT) on a pre-trained LM on the downstream task. We then generate node embeddings using the last hidden states of finetuned LM.
  arXiv  Detail & Related papers  (2023-08-03T07:00:04Z)
• Pitfalls in Link Prediction with Graph Neural Networks: Understanding the Impact of Target-link Inclusion & Better Practices [28.88423949622]
  Graph Neural Networks (GNNs) are remarkably successful in a variety of high-impact applications. In link prediction, the common practices of including the edges being predicted in the graph at training and/or test have outsized impact on the performance of low-degree nodes. We introduce an effective and efficient GNN training framework, SpotTarget, which leverages our insight on low-degree nodes.
  arXiv  Detail & Related papers  (2023-06-01T16:56:04Z)
• DEGREE: Decomposition Based Explanation For Graph Neural Networks [55.38873296761104]
  We propose DEGREE to provide a faithful explanation for GNN predictions. By decomposing the information generation and aggregation mechanism of GNNs, DEGREE allows tracking the contributions of specific components of the input graph to the final prediction. We also design a subgraph level interpretation algorithm to reveal complex interactions between graph nodes that are overlooked by previous methods.
  arXiv  Detail & Related papers  (2023-05-22T10:29:52Z)
• Simple yet Effective Gradient-Free Graph Convolutional Networks [20.448409424929604]
  Linearized Graph Neural Networks (GNNs) have attracted great attention in recent years for graph representation learning. In this paper, we relate over-smoothing with the vanishing gradient phenomenon and craft a gradient-free training framework. Our methods achieve better and more stable performances on node classification tasks with varying depths and cost much less training time.
  arXiv  Detail & Related papers  (2023-02-01T11:00:24Z)
• Neural Graph Matching for Pre-training Graph Neural Networks [72.32801428070749]
  Graph neural networks (GNNs) have been shown powerful capacity at modeling structural data. We present a novel Graph Matching based GNN Pre-Training framework, called GMPT. The proposed method can be applied to fully self-supervised pre-training and coarse-grained supervised pre-training.
  arXiv  Detail & Related papers  (2022-03-03T09:53:53Z)
• Addressing Over-Smoothing in Graph Neural Networks via Deep Supervision [13.180922099929765]
  Deep graph neural networks (GNNs) suffer from over-smoothing when the number of layers increases. We propose DSGNNs enhanced with deep supervision where representations learned at all layers are used for training. We show that DSGNNs are resilient to over-smoothing and can outperform competitive benchmarks on node and graph property prediction problems.
  arXiv  Detail & Related papers  (2022-02-25T06:05:55Z)
• Investigating Transfer Learning in Graph Neural Networks [2.320417845168326]
  Graph neural networks (GNNs) build on the success of deep learning models by extending them for use in graph spaces. transfer learning has proven extremely successful for traditional deep learning problems: resulting in faster training and improved performance. This research demonstrates that transfer learning is effective with GNNs, and describes how source tasks and the choice of GNN impact the ability to learn generalisable knowledge.
  arXiv  Detail & Related papers  (2022-02-01T20:33:15Z)
• Towards Unsupervised Deep Graph Structure Learning [67.58720734177325]
  We propose an unsupervised graph structure learning paradigm, where the learned graph topology is optimized by data itself without any external guidance. Specifically, we generate a learning target from the original data as an "anchor graph", and use a contrastive loss to maximize the agreement between the anchor graph and the learned graph.
  arXiv  Detail & Related papers  (2022-01-17T11:57:29Z)
• Attentive Graph Neural Networks for Few-Shot Learning [74.01069516079379]
  Graph Neural Networks (GNN) has demonstrated the superior performance in many challenging applications, including the few-shot learning tasks. Despite its powerful capacity to learn and generalize the model from few samples, GNN usually suffers from severe over-fitting and over-smoothing as the model becomes deep. We propose a novel Attentive GNN to tackle these challenges, by incorporating a triple-attention mechanism.
  arXiv  Detail & Related papers  (2020-07-14T07:43:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.