Fast and Continual Knowledge Graph Embedding via Incremental LoRA

• URL: http://arxiv.org/abs/2407.05705v1
• Date: Mon, 8 Jul 2024 08:07:13 GMT
• Title: Fast and Continual Knowledge Graph Embedding via Incremental LoRA
• Authors: Jiajun Liu, Wenjun Ke, Peng Wang, Jiahao Wang, Jinhua Gao, Ziyu Shang, Guozheng Li, Zijie Xu, Ke Ji, Yining Li,
• Abstract summary: Continual Knowledge Graph Embedding aims to efficiently learn new knowledge and simultaneously preserve old knowledge. We propose a fast CKGE framework (model) incorporating an incremental low-rank adapter (mec) mechanism to efficiently acquire new knowledge. We conduct experiments on four public datasets and two new datasets with a larger initial scale.
• Score: 20.624310261539694
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Continual Knowledge Graph Embedding (CKGE) aims to efficiently learn new knowledge and simultaneously preserve old knowledge. Dominant approaches primarily focus on alleviating catastrophic forgetting of old knowledge but neglect efficient learning for the emergence of new knowledge. However, in real-world scenarios, knowledge graphs (KGs) are continuously growing, which brings a significant challenge to fine-tuning KGE models efficiently. To address this issue, we propose a fast CKGE framework (\model), incorporating an incremental low-rank adapter (\mec) mechanism to efficiently acquire new knowledge while preserving old knowledge. Specifically, to mitigate catastrophic forgetting, \model\ isolates and allocates new knowledge to specific layers based on the fine-grained influence between old and new KGs. Subsequently, to accelerate fine-tuning, \model\ devises an efficient \mec\ mechanism, which embeds the specific layers into incremental low-rank adapters with fewer training parameters. Moreover, \mec\ introduces adaptive rank allocation, which makes the LoRA aware of the importance of entities and adjusts its rank scale adaptively. We conduct experiments on four public datasets and two new datasets with a larger initial scale. Experimental results demonstrate that \model\ can reduce training time by 34\%-49\% while still achieving competitive link prediction performance against state-of-the-art models on four public datasets (average MRR score of 21.0\% vs. 21.1\%).Meanwhile, on two newly constructed datasets, \model\ saves 51\%-68\% training time and improves link prediction performance by 1.5\%.

Related papers

• Towards Continual Knowledge Graph Embedding via Incremental Distillation [12.556752486002356]
  Traditional knowledge graph embedding (KGE) methods typically require preserving the entire knowledge graph (KG) with significant training costs when new knowledge emerges. This paper proposes a competitive method for CKGE based on incremental distillation (IncDE), which considers the full use of the explicit graph structure in KGs.
  arXiv  Detail & Related papers  (2024-05-07T16:16:00Z)
• Dense Network Expansion for Class Incremental Learning [61.00081795200547]
  State-of-the-art approaches use a dynamic architecture based on network expansion (NE), in which a task expert is added per task. A new NE method, dense network expansion (DNE), is proposed to achieve a better trade-off between accuracy and model complexity. It outperforms the previous SOTA methods by a margin of 4% in terms of accuracy, with similar or even smaller model scale.
  arXiv  Detail & Related papers  (2023-03-22T16:42:26Z)
• Directed Acyclic Graph Factorization Machines for CTR Prediction via Knowledge Distillation [65.62538699160085]
  We propose a Directed Acyclic Graph Factorization Machine (KD-DAGFM) to learn the high-order feature interactions from existing complex interaction models for CTR prediction via Knowledge Distillation. KD-DAGFM achieves the best performance with less than 21.5% FLOPs of the state-of-the-art method on both online and offline experiments.
  arXiv  Detail & Related papers  (2022-11-21T03:09:42Z)
• Augmented Bilinear Network for Incremental Multi-Stock Time-Series Classification [83.23129279407271]
  We propose a method to efficiently retain the knowledge available in a neural network pre-trained on a set of securities. In our method, the prior knowledge encoded in a pre-trained neural network is maintained by keeping existing connections fixed. This knowledge is adjusted for the new securities by a set of augmented connections, which are optimized using the new data.
  arXiv  Detail & Related papers  (2022-07-23T18:54:10Z)
• Contextual Squeeze-and-Excitation for Efficient Few-Shot Image Classification [57.36281142038042]
  We present a new adaptive block called Contextual Squeeze-and-Excitation (CaSE) that adjusts a pretrained neural network on a new task to significantly improve performance. We also present a new training protocol based on Coordinate-Descent called UpperCaSE that exploits meta-trained CaSE blocks and fine-tuning routines for efficient adaptation.
  arXiv  Detail & Related papers  (2022-06-20T15:25:08Z)
• CascadER: Cross-Modal Cascading for Knowledge Graph Link Prediction [22.96768147978534]
  We propose a tiered ranking architecture CascadER to maintain the ranking accuracy of full ensembling while improving efficiency considerably. CascadER uses LMs to rerank the outputs of more efficient base KGEs, relying on an adaptive subset selection scheme aimed at invoking the LMs minimally while maximizing accuracy gain over the KGE. Our empirical analyses reveal that diversity of models across modalities and preservation of individual models' confidence signals help explain the effectiveness of CascadER.
  arXiv  Detail & Related papers  (2022-05-16T22:55:45Z)
• Swift and Sure: Hardness-aware Contrastive Learning for Low-dimensional Knowledge Graph Embeddings [20.693275018860287]
  We propose a novel KGE training framework called Hardness-aware Low-dimensional Embedding (HaLE) In the limited training time, HaLE can effectively improve the performance and training speed of KGE models. The HaLE-trained models can obtain a high prediction accuracy after training few minutes and are competitive compared to the state-of-the-art models.
  arXiv  Detail & Related papers  (2022-01-03T10:25:10Z)
• Causal Incremental Graph Convolution for Recommender System Retraining [89.25922726558875]
  Real-world recommender system needs to be regularly retrained to keep with the new data. In this work, we consider how to efficiently retrain graph convolution network (GCN) based recommender models.
  arXiv  Detail & Related papers  (2021-08-16T04:20:09Z)
• Class-incremental Learning with Rectified Feature-Graph Preservation [24.098892115785066]
  A central theme of this paper is to learn new classes that arrive in sequential phases over time. We propose a weighted-Euclidean regularization for old knowledge preservation. We show how it can work with binary cross-entropy to increase class separation for effective learning of new classes.
  arXiv  Detail & Related papers  (2020-12-15T07:26:04Z)
• iTAML: An Incremental Task-Agnostic Meta-learning Approach [123.10294801296926]
  Humans can continuously learn new knowledge as their experience grows. Previous learning in deep neural networks can quickly fade out when they are trained on a new task. We introduce a novel meta-learning approach that seeks to maintain an equilibrium between all encountered tasks.
  arXiv  Detail & Related papers  (2020-03-25T21:42:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.