Reducing Catastrophic Forgetting in Online Class Incremental Learning Using Self-Distillation

• URL: http://arxiv.org/abs/2409.11329v1
• Date: Tue, 17 Sep 2024 16:26:33 GMT
• Title: Reducing Catastrophic Forgetting in Online Class Incremental Learning Using Self-Distillation
• Authors: Kotaro Nagata, Hiromu Ono, Kazuhiro Hotta,
• Abstract summary: In continual learning, previous knowledge is forgotten when a model learns new tasks. In this paper, we tried to solve this problem by acquiring transferable knowledge through self-distillation. Our proposed method outperformed conventional methods by experiments on CIFAR10, CIFAR100, and MiniimageNet datasets.
• Score: 3.8506666685467343
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In continual learning, there is a serious problem of catastrophic forgetting, in which previous knowledge is forgotten when a model learns new tasks. Various methods have been proposed to solve this problem. Replay methods which replay data from previous tasks in later training, have shown good accuracy. However, replay methods have a generalizability problem from a limited memory buffer. In this paper, we tried to solve this problem by acquiring transferable knowledge through self-distillation using highly generalizable output in shallow layer as a teacher. Furthermore, when we deal with a large number of classes or challenging data, there is a risk of learning not converging and not experiencing overfitting. Therefore, we attempted to achieve more efficient and thorough learning by prioritizing the storage of easily misclassified samples through a new method of memory update. We confirmed that our proposed method outperformed conventional methods by experiments on CIFAR10, CIFAR100, and MiniimageNet datasets.

Related papers

• TEAL: New Selection Strategy for Small Buffers in Experience Replay Class Incremental Learning [7.627299398469962]
  We introduce TEAL, a novel approach to populate the memory with exemplars. We show that TEAL improves the average accuracy of the SOTA method XDER as well as ER and ER-ACE on several image recognition benchmarks.
  arXiv  Detail & Related papers  (2024-06-30T12:09:08Z)
• Enhancing Consistency and Mitigating Bias: A Data Replay Approach for Incremental Learning [100.7407460674153]
  Deep learning systems are prone to catastrophic forgetting when learning from a sequence of tasks. To mitigate the problem, a line of methods propose to replay the data of experienced tasks when learning new tasks. However, it is not expected in practice considering the memory constraint or data privacy issue. As a replacement, data-free data replay methods are proposed by inverting samples from the classification model.
  arXiv  Detail & Related papers  (2024-01-12T12:51:12Z)
• Fine-Grained Knowledge Selection and Restoration for Non-Exemplar Class Incremental Learning [64.14254712331116]
  Non-exemplar class incremental learning aims to learn both the new and old tasks without accessing any training data from the past. We propose a novel framework of fine-grained knowledge selection and restoration.
  arXiv  Detail & Related papers  (2023-12-20T02:34:11Z)
• Continual Learning via Manifold Expansion Replay [36.27348867557826]
  Catastrophic forgetting is a major challenge to continual learning. We propose a novel replay strategy called Replay Manifold Expansion (MaER) We show that the proposed method significantly improves the accuracy in continual learning setup, outperforming the state of the arts.
  arXiv  Detail & Related papers  (2023-10-12T05:09:27Z)
• Detachedly Learn a Classifier for Class-Incremental Learning [11.865788374587734]
  We present an analysis that the failure of vanilla experience replay (ER) comes from unnecessary re-learning of previous tasks and incompetence to distinguish current task from the previous ones. We propose a novel replay strategy task-aware experience replay. Experimental results show our method outperforms current state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-02-23T01:35:44Z)
• A Memory Transformer Network for Incremental Learning [64.0410375349852]
  We study class-incremental learning, a training setup in which new classes of data are observed over time for the model to learn from. Despite the straightforward problem formulation, the naive application of classification models to class-incremental learning results in the "catastrophic forgetting" of previously seen classes. One of the most successful existing methods has been the use of a memory of exemplars, which overcomes the issue of catastrophic forgetting by saving a subset of past data into a memory bank and utilizing it to prevent forgetting when training future tasks.
  arXiv  Detail & Related papers  (2022-10-10T08:27:28Z)
• Few-Shot Class-Incremental Learning via Entropy-Regularized Data-Free Replay [52.251188477192336]
  Few-shot class-incremental learning (FSCIL) has been proposed aiming to enable a deep learning system to incrementally learn new classes with limited data. We show through empirical results that adopting the data replay is surprisingly favorable. We propose using data-free replay that can synthesize data by a generator without accessing real data.
  arXiv  Detail & Related papers  (2022-07-22T17:30:51Z)
• Relational Experience Replay: Continual Learning by Adaptively Tuning Task-wise Relationship [54.73817402934303]
  We propose Experience Continual Replay (ERR), a bi-level learning framework to adaptively tune task-wise to achieve a better stability plasticity' tradeoff. ERR can consistently improve the performance of all baselines and surpass current state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-12-31T12:05:22Z)
• Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learning [73.24988226158497]
  We consider the high-impact problem of Data-Free Class-Incremental Learning (DFCIL) We propose a novel incremental distillation strategy for DFCIL, contributing a modified cross-entropy training and importance-weighted feature distillation. Our method results in up to a 25.1% increase in final task accuracy (absolute difference) compared to SOTA DFCIL methods for common class-incremental benchmarks.
  arXiv  Detail & Related papers  (2021-06-17T17:56:08Z)
• Self-Supervised Learning Aided Class-Incremental Lifelong Learning [17.151579393716958]
  We study the issue of catastrophic forgetting in class-incremental learning (Class-IL) In training procedure of Class-IL, as the model has no knowledge about following tasks, it would only extract features necessary for tasks learned so far, whose information is insufficient for joint classification. We propose to combine self-supervised learning, which can provide effective representations without requiring labels, with Class-IL to partly get around this problem.
  arXiv  Detail & Related papers  (2020-06-10T15:15:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.