Feature Expansion and enhanced Compression for Class Incremental Learning

• URL: http://arxiv.org/abs/2405.08038v1
• Date: Mon, 13 May 2024 06:57:18 GMT
• Title: Feature Expansion and enhanced Compression for Class Incremental Learning
• Authors: Quentin Ferdinand, Gilles Le Chenadec, Benoit Clement, Panagiotis Papadakis, Quentin Oliveau,
• Abstract summary: We propose a new algorithm that enhances the compression of previous class knowledge by cutting and mixing patches of previous class samples with the new images during compression. We show that this new data augmentation reduces catastrophic forgetting by specifically targeting past class information and improving its compression.
• Score: 3.3425792454347616
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Class incremental learning consists in training discriminative models to classify an increasing number of classes over time. However, doing so using only the newly added class data leads to the known problem of catastrophic forgetting of the previous classes. Recently, dynamic deep learning architectures have been shown to exhibit a better stability-plasticity trade-off by dynamically adding new feature extractors to the model in order to learn new classes followed by a compression step to scale the model back to its original size, thus avoiding a growing number of parameters. In this context, we propose a new algorithm that enhances the compression of previous class knowledge by cutting and mixing patches of previous class samples with the new images during compression using our Rehearsal-CutMix method. We show that this new data augmentation reduces catastrophic forgetting by specifically targeting past class information and improving its compression. Extensive experiments performed on the CIFAR and ImageNet datasets under diverse incremental learning evaluation protocols demonstrate that our approach consistently outperforms the state-of-the-art . The code will be made available upon publication of our work.

Related papers

• CEAT: Continual Expansion and Absorption Transformer for Non-Exemplar Class-Incremental Learning [34.59310641291726]
  In real-world applications, dynamic scenarios require the models to possess the capability to learn new tasks continuously without forgetting the old knowledge. We propose a new architecture, named continual expansion and absorption transformer(CEAT) The model can learn the novel knowledge by extending the expanded-fusion layers in parallel with the frozen previous parameters. To improve the learning ability of the model, we designed a novel prototype contrastive loss to reduce the overlap between old and new classes in the feature space.
  arXiv  Detail & Related papers  (2024-03-11T12:40:12Z)
• Class-Incremental Learning using Diffusion Model for Distillation and Replay [5.0977390531431634]
  Class-incremental learning aims to learn new classes in an incremental fashion without forgetting the previously learned ones. We propose the use of a pretrained Stable Diffusion model as a source of additional data for class-incremental learning.
  arXiv  Detail & Related papers  (2023-06-30T11:23:49Z)
• Non-exemplar Class-incremental Learning by Random Auxiliary Classes Augmentation and Mixed Features [37.51376572211081]
  Non-exemplar class-incremental learning refers to classifying new and old classes without storing samples of old classes. We propose an effective non-exemplar method called RAMF consisting of Random Auxiliary classes augmentation and Mixed Feature.
  arXiv  Detail & Related papers  (2023-04-16T06:33:43Z)
• FOSTER: Feature Boosting and Compression for Class-Incremental Learning [52.603520403933985]
  Deep neural networks suffer from catastrophic forgetting when learning new categories. We propose a novel two-stage learning paradigm FOSTER, empowering the model to learn new categories adaptively.
  arXiv  Detail & Related papers  (2022-04-10T11:38:33Z)
• Self-Supervised Class Incremental Learning [51.62542103481908]
  Existing Class Incremental Learning (CIL) methods are based on a supervised classification framework sensitive to data labels. When updating them based on the new class data, they suffer from catastrophic forgetting: the model cannot discern old class data clearly from the new. In this paper, we explore the performance of Self-Supervised representation learning in Class Incremental Learning (SSCIL) for the first time.
  arXiv  Detail & Related papers  (2021-11-18T06:58:19Z)
• Subspace Regularizers for Few-Shot Class Incremental Learning [26.372024890126408]
  We present a new family of subspace regularization schemes that encourage weight vectors for new classes to lie close to the subspace spanned by the weights of existing classes. Our results show that simple geometric regularization of class representations offers an effective tool for continual learning.
  arXiv  Detail & Related papers  (2021-10-13T22:19:53Z)
• Powerpropagation: A sparsity inducing weight reparameterisation [65.85142037667065]
  We introduce Powerpropagation, a new weight- parameterisation for neural networks that leads to inherently sparse models. Models trained in this manner exhibit similar performance, but have a distribution with markedly higher density at zero, allowing more parameters to be pruned safely. Here, we combine Powerpropagation with a traditional weight-pruning technique as well as recent state-of-the-art sparse-to-sparse algorithms, showing superior performance on the ImageNet benchmark.
  arXiv  Detail & Related papers  (2021-10-01T10:03:57Z)
• Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learning [73.24988226158497]
  We consider the high-impact problem of Data-Free Class-Incremental Learning (DFCIL) We propose a novel incremental distillation strategy for DFCIL, contributing a modified cross-entropy training and importance-weighted feature distillation. Our method results in up to a 25.1% increase in final task accuracy (absolute difference) compared to SOTA DFCIL methods for common class-incremental benchmarks.
  arXiv  Detail & Related papers  (2021-06-17T17:56:08Z)
• Half-Real Half-Fake Distillation for Class-Incremental Semantic Segmentation [84.1985497426083]
  convolutional neural networks are ill-equipped for incremental learning. New classes are available but the initial training data is not retained. We try to address this issue by "inverting" the trained segmentation network to synthesize input images starting from random noise.
  arXiv  Detail & Related papers  (2021-04-02T03:47:16Z)
• On the Exploration of Incremental Learning for Fine-grained Image Retrieval [45.48333682748607]
  We consider the problem of fine-grained image retrieval in an incremental setting, when new categories are added over time. We propose an incremental learning method to mitigate retrieval performance degradation caused by the forgetting issue. Our method effectively mitigates the catastrophic forgetting on the original classes while achieving high performance on the new classes.
  arXiv  Detail & Related papers  (2020-10-15T21:07:44Z)
• Memory-Efficient Incremental Learning Through Feature Adaptation [71.1449769528535]
  We introduce an approach for incremental learning that preserves feature descriptors of training images from previously learned classes. Keeping the much lower-dimensional feature embeddings of images reduces the memory footprint significantly. Experimental results show that our method achieves state-of-the-art classification accuracy in incremental learning benchmarks.
  arXiv  Detail & Related papers  (2020-04-01T21:16:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.