Selecting Related Knowledge via Efficient Channel Attention for Online Continual Learning

• URL: http://arxiv.org/abs/2209.04212v1
• Date: Fri, 9 Sep 2022 09:59:54 GMT
• Title: Selecting Related Knowledge via Efficient Channel Attention for Online Continual Learning
• Authors: Ya-nan Han, Jian-wei Liu
• Abstract summary: We propose a new framework, named Selecting Related Knowledge for Online Continual Learning (SRKOCL) Our model also combines experience replay and knowledge distillation to circumvent the catastrophic forgetting.
• Score: 4.109784267309124
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Continual learning aims to learn a sequence of tasks by leveraging the knowledge acquired in the past in an online-learning manner while being able to perform well on all previous tasks, this ability is crucial to the artificial intelligence (AI) system, hence continual learning is more suitable for most real-word and complex applicative scenarios compared to the traditional learning pattern. However, the current models usually learn a generic representation base on the class label on each task and an effective strategy is selected to avoid catastrophic forgetting. We postulate that selecting the related and useful parts only from the knowledge obtained to perform each task is more effective than utilizing the whole knowledge. Based on this fact, in this paper we propose a new framework, named Selecting Related Knowledge for Online Continual Learning (SRKOCL), which incorporates an additional efficient channel attention mechanism to pick the particular related knowledge for every task. Our model also combines experience replay and knowledge distillation to circumvent the catastrophic forgetting. Finally, extensive experiments are conducted on different benchmarks and the competitive experimental results demonstrate that our proposed SRKOCL is a promised approach against the state-of-the-art.

Related papers

• A Unified and General Framework for Continual Learning [58.72671755989431]
  Continual Learning (CL) focuses on learning from dynamic and changing data distributions while retaining previously acquired knowledge. Various methods have been developed to address the challenge of catastrophic forgetting, including regularization-based, Bayesian-based, and memory-replay-based techniques. This research aims to bridge this gap by introducing a comprehensive and overarching framework that encompasses and reconciles these existing methodologies.
  arXiv  Detail & Related papers  (2024-03-20T02:21:44Z)
• Quiz-based Knowledge Tracing [61.9152637457605]
  Knowledge tracing aims to assess individuals' evolving knowledge states according to their learning interactions. QKT achieves state-of-the-art performance compared to existing methods.
  arXiv  Detail & Related papers  (2023-04-05T12:48:42Z)
• Online Continual Learning via the Knowledge Invariant and Spread-out Properties [4.109784267309124]
  Key challenge in continual learning is catastrophic forgetting. We propose a new method, named Online Continual Learning via the Knowledge Invariant and Spread-out Properties (OCLKISP) We empirically evaluate our proposed method on four popular benchmarks for continual learning: Split CIFAR 100, Split SVHN, Split CUB200 and Split Tiny-Image-Net.
  arXiv  Detail & Related papers  (2023-02-02T04:03:38Z)
• Unveiling the Tapestry: the Interplay of Generalization and Forgetting in Continual Learning [18.61040106667249]
  In AI, generalization refers to a model's ability to perform well on out-of-distribution data related to a given task, beyond the data it was trained on. Continual learning methods often include mechanisms to mitigate catastrophic forgetting, ensuring that knowledge from earlier tasks is retained. We introduce a simple and effective technique known as Shape-Texture Consistency Regularization (STCR), which caters to continual learning.
  arXiv  Detail & Related papers  (2022-11-21T04:36:24Z)
• Hierarchically Structured Task-Agnostic Continual Learning [0.0]
  We take a task-agnostic view of continual learning and develop a hierarchical information-theoretic optimality principle. We propose a neural network layer, called the Mixture-of-Variational-Experts layer, that alleviates forgetting by creating a set of information processing paths. Our approach can operate in a task-agnostic way, i.e., it does not require task-specific knowledge, as is the case with many existing continual learning algorithms.
  arXiv  Detail & Related papers  (2022-11-14T19:53:15Z)
• Toward Sustainable Continual Learning: Detection and Knowledge Repurposing of Similar Tasks [31.095642850920385]
  We introduce a paradigm where the continual learner gets a sequence of mixed similar and dissimilar tasks. We propose a new continual learning framework that uses a task similarity detection function that does not require additional learning. Our experiments show that the proposed framework performs competitively on widely used computer vision benchmarks.
  arXiv  Detail & Related papers  (2022-10-11T19:35:30Z)
• A Closer Look at Knowledge Distillation with Features, Logits, and Gradients [81.39206923719455]
  Knowledge distillation (KD) is a substantial strategy for transferring learned knowledge from one neural network model to another. This work provides a new perspective to motivate a set of knowledge distillation strategies by approximating the classical KL-divergence criteria with different knowledge sources. Our analysis indicates that logits are generally a more efficient knowledge source and suggests that having sufficient feature dimensions is crucial for the model design.
  arXiv  Detail & Related papers  (2022-03-18T21:26:55Z)
• Continual Prompt Tuning for Dialog State Tracking [58.66412648276873]
  A desirable dialog system should be able to continually learn new skills without forgetting old ones. We present Continual Prompt Tuning, a parameter-efficient framework that not only avoids forgetting but also enables knowledge transfer between tasks.
  arXiv  Detail & Related papers  (2022-03-13T13:22:41Z)
• Self-Attention Meta-Learner for Continual Learning [5.979373021392084]
  Self-Attention Meta-Learner (SAM) learns a prior knowledge for continual learning that permits learning a sequence of tasks. SAM incorporates an attention mechanism that learns to select the particular relevant representation for each future task. We evaluate the proposed method on the Split CIFAR-10/100 and Split MNIST benchmarks in the task inference.
  arXiv  Detail & Related papers  (2021-01-28T17:35:04Z)
• Towards a Universal Continuous Knowledge Base [49.95342223987143]
  We propose a method for building a continuous knowledge base that can store knowledge imported from multiple neural networks. Experiments on text classification show promising results. We import the knowledge from multiple models to the knowledge base, from which the fused knowledge is exported back to a single model.
  arXiv  Detail & Related papers  (2020-12-25T12:27:44Z)
• Bilevel Continual Learning [76.50127663309604]
  We present a novel framework of continual learning named "Bilevel Continual Learning" (BCL) Our experiments on continual learning benchmarks demonstrate the efficacy of the proposed BCL compared to many state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-07-30T16:00:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.