REAL: Representation Enhanced Analytic Learning for Exemplar-free Class-incremental Learning

• URL: http://arxiv.org/abs/2403.13522v1
• Date: Wed, 20 Mar 2024 11:48:10 GMT
• Title: REAL: Representation Enhanced Analytic Learning for Exemplar-free Class-incremental Learning
• Authors: Run He, Huiping Zhuang, Di Fang, Yizhu Chen, Kai Tong, Cen Chen,
• Abstract summary: We propose a representation enhanced analytic learning (REAL) for Exemplar-free class-incremental learning (EFCIL) The REAL constructs a dual-stream base pretraining (DS-BPT) and a representation enhancing distillation (RED) process to enhance the representation of the extractor. Our method addresses the issue of insufficient discriminability in representations of unseen data caused by a frozen backbone in the existing AL-based CIL.
• Score: 12.197327462627912
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Exemplar-free class-incremental learning (EFCIL) aims to mitigate catastrophic forgetting in class-incremental learning without available historical data. Compared with its counterpart (replay-based CIL) that stores historical samples, the EFCIL suffers more from forgetting issues under the exemplar-free constraint. In this paper, inspired by the recently developed analytic learning (AL) based CIL, we propose a representation enhanced analytic learning (REAL) for EFCIL. The REAL constructs a dual-stream base pretraining (DS-BPT) and a representation enhancing distillation (RED) process to enhance the representation of the extractor. The DS-BPT pretrains model in streams of both supervised learning and self-supervised contrastive learning (SSCL) for base knowledge extraction. The RED process distills the supervised knowledge to the SSCL pretrained backbone and facilitates a subsequent AL-basd CIL that converts the CIL to a recursive least-square problem. Our method addresses the issue of insufficient discriminability in representations of unseen data caused by a frozen backbone in the existing AL-based CIL. Empirical results on various datasets including CIFAR-100, ImageNet-100 and ImageNet-1k, demonstrate that our REAL outperforms the state-of-the-arts in EFCIL, and achieves comparable or even more superior performance compared with the replay-based methods.

Related papers

• DS-AL: A Dual-Stream Analytic Learning for Exemplar-Free Class-Incremental Learning [27.585993693155217]
  Class-incremental learning (CIL) under an exemplar-free constraint has presented a significant challenge. We propose a Dual-Stream Analytic Learning (DS-AL) approach to solve the exemplar-free CIL problem. Empirical results demonstrate that the DS-AL, despite being an exemplar-free technique, delivers performance comparable with or better than that of replay-based methods.
  arXiv  Detail & Related papers  (2024-03-26T09:04:18Z)
• Exemplar-Free Class Incremental Learning via Incremental Representation [26.759108983223115]
  We propose a textbfsimple Incremental Representation (IR) framework for efCIL without constructing old pseudo-features. IR utilizes dataset augmentation to cover a suitable feature space and prevents the model from forgetting by using a single L2 space maintenance loss.
  arXiv  Detail & Related papers  (2024-03-24T16:29:50Z)
• Learning Prompt with Distribution-Based Feature Replay for Few-Shot Class-Incremental Learning [56.29097276129473]
  We propose a simple yet effective framework, named Learning Prompt with Distribution-based Feature Replay (LP-DiF) To prevent the learnable prompt from forgetting old knowledge in the new session, we propose a pseudo-feature replay approach. When progressing to a new session, pseudo-features are sampled from old-class distributions combined with training images of the current session to optimize the prompt.
  arXiv  Detail & Related papers  (2024-01-03T07:59:17Z)
• Boosting Differentiable Causal Discovery via Adaptive Sample Reweighting [62.23057729112182]
  Differentiable score-based causal discovery methods learn a directed acyclic graph from observational data. We propose a model-agnostic framework to boost causal discovery performance by dynamically learning the adaptive weights for the Reweighted Score function, ReScore.
  arXiv  Detail & Related papers  (2023-03-06T14:49:59Z)
• Learning Deep Representations via Contrastive Learning for Instance Retrieval [11.736450745549792]
  This paper makes the first attempt that tackles the problem using instance-discrimination based contrastive learning (CL) In this work, we approach this problem by exploring the capability of deriving discriminative representations from pre-trained and fine-tuned CL models.
  arXiv  Detail & Related papers  (2022-09-28T04:36:34Z)
• CCLF: A Contrastive-Curiosity-Driven Learning Framework for Sample-Efficient Reinforcement Learning [56.20123080771364]
  We develop a model-agnostic Contrastive-Curiosity-Driven Learning Framework (CCLF) for reinforcement learning. CCLF fully exploit sample importance and improve learning efficiency in a self-supervised manner. We evaluate this approach on the DeepMind Control Suite, Atari, and MiniGrid benchmarks.
  arXiv  Detail & Related papers  (2022-05-02T14:42:05Z)
• R-DFCIL: Relation-Guided Representation Learning for Data-Free Class Incremental Learning [64.7996065569457]
  Class-Incremental Learning (CIL) struggles with catastrophic forgetting when learning new knowledge. Recent DFCIL works introduce techniques such as model inversion to synthesize data for previous classes, they fail to overcome forgetting due to the severe domain gap between the synthetic and real data. This paper proposes relation-guided representation learning (RRL) for DFCIL, dubbed R-DFCIL.
  arXiv  Detail & Related papers  (2022-03-24T14:54:15Z)
• Self-Supervised Class Incremental Learning [51.62542103481908]
  Existing Class Incremental Learning (CIL) methods are based on a supervised classification framework sensitive to data labels. When updating them based on the new class data, they suffer from catastrophic forgetting: the model cannot discern old class data clearly from the new. In this paper, we explore the performance of Self-Supervised representation learning in Class Incremental Learning (SSCIL) for the first time.
  arXiv  Detail & Related papers  (2021-11-18T06:58:19Z)
• No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Data [78.69828864672978]
  A central challenge in training classification models in the real-world federated system is learning with non-IID data. We propose a novel and simple algorithm called Virtual Representations (CCVR), which adjusts the classifier using virtual representations sampled from an approximated ssian mixture model. Experimental results demonstrate that CCVR state-of-the-art performance on popular federated learning benchmarks including CIFAR-10, CIFAR-100, and CINIC-10.
  arXiv  Detail & Related papers  (2021-06-09T12:02:29Z)
• IB-DRR: Incremental Learning with Information-Back Discrete Representation Replay [4.8666876477091865]
  Incremental learning aims to enable machine learning models to continuously acquire new knowledge given new classes. Saving a subset of training samples of previously seen classes in the memory and replaying them during new training phases is proven to be an efficient and effective way to fulfil this aim. However, finding a trade-off between the model performance and the number of samples to save for each class is still an open problem for replay-based incremental learning.
  arXiv  Detail & Related papers  (2021-04-21T15:32:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.