History Repeats: Overcoming Catastrophic Forgetting For Event-Centric Temporal Knowledge Graph Completion

• URL: http://arxiv.org/abs/2305.18675v1
• Date: Tue, 30 May 2023 01:21:36 GMT
• Title: History Repeats: Overcoming Catastrophic Forgetting For Event-Centric Temporal Knowledge Graph Completion
• Authors: Mehrnoosh Mirtaheri, Mohammad Rostami, Aram Galstyan
• Abstract summary: Temporal knowledge graph (TKG) completion models rely on having access to the entire graph during training. TKG data is often received incrementally as events unfold, leading to a dynamic non-stationary data distribution over time. We propose a general continual training framework that is applicable to any TKG completion method.
• Score: 33.38304336898247
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Temporal knowledge graph (TKG) completion models typically rely on having access to the entire graph during training. However, in real-world scenarios, TKG data is often received incrementally as events unfold, leading to a dynamic non-stationary data distribution over time. While one could incorporate fine-tuning to existing methods to allow them to adapt to evolving TKG data, this can lead to forgetting previously learned patterns. Alternatively, retraining the model with the entire updated TKG can mitigate forgetting but is computationally burdensome. To address these challenges, we propose a general continual training framework that is applicable to any TKG completion method, and leverages two key ideas: (i) a temporal regularization that encourages repurposing of less important model parameters for learning new knowledge, and (ii) a clustering-based experience replay that reinforces the past knowledge by selectively preserving only a small portion of the past data. Our experimental results on widely used event-centric TKG datasets demonstrate the effectiveness of our proposed continual training framework in adapting to new events while reducing catastrophic forgetting. Further, we perform ablation studies to show the effectiveness of each component of our proposed framework. Finally, we investigate the relation between the memory dedicated to experience replay and the benefit gained from our clustering-based sampling strategy.

Related papers

• Dynamic Loss-Based Sample Reweighting for Improved Large Language Model Pretraining [55.262510814326035]
  Existing reweighting strategies primarily focus on group-level data importance. We introduce novel algorithms for dynamic, instance-level data reweighting. Our framework allows us to devise reweighting strategies deprioritizing redundant or uninformative data.
  arXiv  Detail & Related papers  (2025-02-10T17:57:15Z)
• Towards Pattern-aware Data Augmentation for Temporal Knowledge Graph Completion [18.51546761241817]
  We introduce Booster, the first data augmentation strategy for temporal knowledge graphs. We propose a hierarchical scoring algorithm based on triadic closures within TKGs. We also propose a two-stage training approach to identify samples that deviate from the model's preferred patterns.
  arXiv  Detail & Related papers  (2024-12-31T03:47:19Z)
• TTVD: Towards a Geometric Framework for Test-Time Adaptation Based on Voronoi Diagram [14.238620530634392]
  Test-time adaptation (TTA) is an emerging scheme used at inference time to address this issue. We study the TTA problem from a geometric point of view. We propose the Test-Time adjustment by Voronoi Diagram guidance (TTVD), a novel framework that leverages the benefits of this geometric property.
  arXiv  Detail & Related papers  (2024-12-10T23:40:07Z)
• Adaptive Retention & Correction: Test-Time Training for Continual Learning [114.5656325514408]
  A common problem in continual learning is the classification layer's bias towards the most recent task. We name our approach Adaptive Retention & Correction (ARC) ARC achieves an average performance increase of 2.7% and 2.6% on the CIFAR-100 and Imagenet-R datasets.
  arXiv  Detail & Related papers  (2024-05-23T08:43:09Z)
• Federated Learning with Projected Trajectory Regularization [65.6266768678291]
  Federated learning enables joint training of machine learning models from distributed clients without sharing their local data. One key challenge in federated learning is to handle non-identically distributed data across the clients. We propose a novel federated learning framework with projected trajectory regularization (FedPTR) for tackling the data issue.
  arXiv  Detail & Related papers  (2023-12-22T02:12:08Z)
• Exploring the Limits of Historical Information for Temporal Knowledge Graph Extrapolation [59.417443739208146]
  We propose a new event forecasting model based on a novel training framework of historical contrastive learning. CENET learns both the historical and non-historical dependency to distinguish the most potential entities. We evaluate our proposed model on five benchmark graphs.
  arXiv  Detail & Related papers  (2023-08-29T03:26:38Z)
• Continual Few-shot Relation Learning via Embedding Space Regularization and Data Augmentation [4.111899441919165]
  It is necessary for the model to learn novel relational patterns with very few labeled data while avoiding catastrophic forgetting of previous task knowledge. We propose a novel method based on embedding space regularization and data augmentation. Our method generalizes to new few-shot tasks and avoids catastrophic forgetting of previous tasks by enforcing extra constraints on the relational embeddings and by adding extra relevant data in a self-supervised manner.
  arXiv  Detail & Related papers  (2022-03-04T05:19:09Z)
• Improved Fine-tuning by Leveraging Pre-training Data: Theory and Practice [52.11183787786718]
  Fine-tuning a pre-trained model on the target data is widely used in many deep learning applications. Recent studies have empirically shown that training from scratch has the final performance that is no worse than this pre-training strategy. We propose a novel selection strategy to select a subset from pre-training data to help improve the generalization on the target task.
  arXiv  Detail & Related papers  (2021-11-24T06:18:32Z)
• Causal Incremental Graph Convolution for Recommender System Retraining [89.25922726558875]
  Real-world recommender system needs to be regularly retrained to keep with the new data. In this work, we consider how to efficiently retrain graph convolution network (GCN) based recommender models.
  arXiv  Detail & Related papers  (2021-08-16T04:20:09Z)
• TIE: A Framework for Embedding-based Incremental Temporal Knowledge Graph Completion [37.76140466390048]
  Reasoning in a temporal knowledge graph (TKG) is a critical task for information retrieval and semantic search. Recent work approaches TKG completion (TKGC) by augmenting the encoder-decoder framework with a time-aware encoding function. We present the Time-aware Incremental Embedding (TIE) framework, which combines TKG representation learning, experience replay, and temporal regularization.
  arXiv  Detail & Related papers  (2021-04-17T01:40:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.