Preserving Earlier Knowledge in Continual Learning with the Help of All Previous Feature Extractors

• URL: http://arxiv.org/abs/2104.13614v1
• Date: Wed, 28 Apr 2021 07:49:24 GMT
• Title: Preserving Earlier Knowledge in Continual Learning with the Help of All Previous Feature Extractors
• Authors: Zhuoyun Li, Changhong Zhong, Sijia Liu, Ruixuan Wang, and Wei-Shi Zheng
• Abstract summary: Continual learning of new knowledge over time is one desirable capability for intelligent systems to recognize more and more classes of objects. We propose a simple yet effective fusion mechanism by including all the previously learned feature extractors into the intelligent model. Experiments on multiple classification tasks show that the proposed approach can effectively reduce the forgetting of old knowledge, achieving state-of-the-art continual learning performance.
• Score: 63.21036904487014
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Continual learning of new knowledge over time is one desirable capability for intelligent systems to recognize more and more classes of objects. Without or with very limited amount of old data stored, an intelligent system often catastrophically forgets previously learned old knowledge when learning new knowledge. Recently, various approaches have been proposed to alleviate the catastrophic forgetting issue. However, old knowledge learned earlier is commonly less preserved than that learned more recently. In order to reduce the forgetting of particularly earlier learned old knowledge and improve the overall continual learning performance, we propose a simple yet effective fusion mechanism by including all the previously learned feature extractors into the intelligent model. In addition, a new feature extractor is included to the model when learning a new set of classes each time, and a feature extractor pruning is also applied to prevent the whole model size from growing rapidly. Experiments on multiple classification tasks show that the proposed approach can effectively reduce the forgetting of old knowledge, achieving state-of-the-art continual learning performance.

Related papers

• Infinite dSprites for Disentangled Continual Learning: Separating Memory Edits from Generalization [36.23065731463065]
  We introduce Infinite dSprites, a parsimonious tool for creating continual classification benchmarks of arbitrary length. We show that over a sufficiently long time horizon, the performance of all major types of continual learning methods deteriorates on this simple benchmark. In a simple setting with direct supervision on the generative factors, we show how learning class-agnostic transformations offers a way to circumvent catastrophic forgetting.
  arXiv  Detail & Related papers  (2023-12-27T22:05:42Z)
• Forgetting before Learning: Utilizing Parametric Arithmetic for Knowledge Updating in Large Language Models [53.52344131257681]
  We propose a new paradigm for fine-tuning called F-Learning, which employs parametric arithmetic to facilitate the forgetting of old knowledge and learning of new knowledge. Experimental results on two publicly available datasets demonstrate that our proposed F-Learning can obviously improve the knowledge updating performance of both full fine-tuning and LoRA fine-tuning.
  arXiv  Detail & Related papers  (2023-11-14T09:12:40Z)
• Adaptively Integrated Knowledge Distillation and Prediction Uncertainty for Continual Learning [71.43841235954453]
  Current deep learning models often suffer from catastrophic forgetting of old knowledge when continually learning new knowledge. Existing strategies to alleviate this issue often fix the trade-off between keeping old knowledge (stability) and learning new knowledge (plasticity)
  arXiv  Detail & Related papers  (2023-01-18T05:36:06Z)
• Learning with Recoverable Forgetting [77.56338597012927]
  Learning wIth Recoverable Forgetting explicitly handles the task- or sample-specific knowledge removal and recovery. Specifically, LIRF brings in two innovative schemes, namely knowledge deposit and withdrawal. We conduct experiments on several datasets, and demonstrate that the proposed LIRF strategy yields encouraging results with gratifying generalization capability.
  arXiv  Detail & Related papers  (2022-07-17T16:42:31Z)
• Continual Learning with Bayesian Model based on a Fixed Pre-trained Feature Extractor [55.9023096444383]
  Current deep learning models are characterised by catastrophic forgetting of old knowledge when learning new classes. Inspired by the process of learning new knowledge in human brains, we propose a Bayesian generative model for continual learning.
  arXiv  Detail & Related papers  (2022-04-28T08:41:51Z)
• Continually Learning Self-Supervised Representations with Projected Functional Regularization [39.92600544186844]
  Recent self-supervised learning methods are able to learn high-quality image representations and are closing the gap with supervised methods. These methods are unable to acquire new knowledge incrementally -- they are, in fact, mostly used only as a pre-training phase with IID data. To prevent forgetting of previous knowledge, we propose the usage of functional regularization.
  arXiv  Detail & Related papers  (2021-12-30T11:59:23Z)
• Discriminative Distillation to Reduce Class Confusion in Continual Learning [57.715862676788156]
  Class confusion may play a role in downgrading the classification performance during continual learning. We propose a discriminative distillation strategy to help the classify well learn the discriminative features between confusing classes.
  arXiv  Detail & Related papers  (2021-08-11T12:46:43Z)
• On Learning the Geodesic Path for Incremental Learning [38.222736913855115]
  Neural networks notoriously suffer from the problem of catastrophic forgetting, the phenomenon of forgetting the past knowledge when acquiring new knowledge. Overcoming catastrophic forgetting is of significant importance to emulate the process of "incremental learning" State-of-the-art techniques for incremental learning make use of knowledge distillation towards preventing catastrophic forgetting.
  arXiv  Detail & Related papers  (2021-04-17T15:26:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.