A Comprehensive Study of Class Incremental Learning Algorithms for Visual Tasks

• URL: http://arxiv.org/abs/2011.01844v4
• Date: Tue, 15 Dec 2020 16:40:55 GMT
• Title: A Comprehensive Study of Class Incremental Learning Algorithms for Visual Tasks
• Authors: Eden Belouadah, Adrian Popescu and Ioannis Kanellos
• Abstract summary: The ability of artificial agents to increment their capabilities when confronted with new data is an open challenge in artificial intelligence. Main challenge is catastrophic forgetting, i.e., the tendency of neural networks to underfit past data when new ones are ingested. We propose a common evaluation framework which is more thorough than existing ones in terms of number of datasets, size of datasets, size of bounded memory and number of incremental states.
• Score: 11.230170401360633
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The ability of artificial agents to increment their capabilities when confronted with new data is an open challenge in artificial intelligence. The main challenge faced in such cases is catastrophic forgetting, i.e., the tendency of neural networks to underfit past data when new ones are ingested. A first group of approaches tackles forgetting by increasing deep model capacity to accommodate new knowledge. A second type of approaches fix the deep model size and introduce a mechanism whose objective is to ensure a good compromise between stability and plasticity of the model. While the first type of algorithms were compared thoroughly, this is not the case for methods which exploit a fixed size model. Here, we focus on the latter, place them in a common conceptual and experimental framework and propose the following contributions: (1) define six desirable properties of incremental learning algorithms and analyze them according to these properties, (2) introduce a unified formalization of the class-incremental learning problem, (3) propose a common evaluation framework which is more thorough than existing ones in terms of number of datasets, size of datasets, size of bounded memory and number of incremental states, (4) investigate the usefulness of herding for past exemplars selection, (5) provide experimental evidence that it is possible to obtain competitive performance without the use of knowledge distillation to tackle catastrophic forgetting and (6) facilitate reproducibility by integrating all tested methods in a common open-source repository. The main experimental finding is that none of the existing algorithms achieves the best results in all evaluated settings. Important differences arise notably if a bounded memory of past classes is allowed or not.

Related papers

• Towards Effective Open-set Graph Class-incremental Learning [4.286860874195651]
  Graph class-incremental learning (GCIL) allows graph neural networks (GNNs) to adapt to evolving graph analytical tasks by incrementally learning new class knowledge.<n>Existing GCIL methods primarily focus on a closed-set assumption, where all test samples are presumed to belong to previously known classes.<n>We propose a more challenging open-set graph class-incremental learning scenario with two intertwined challenges: catastrophic forgetting of old classes, and inadequate open-set recognition.
  arXiv  Detail & Related papers  (2025-07-23T16:51:23Z)
• RESTOR: Knowledge Recovery through Machine Unlearning [71.75834077528305]
  Large language models trained on web-scale corpora can memorize undesirable datapoints. Many machine unlearning methods have been proposed that aim to 'erase' these datapoints from trained models. We propose the RESTOR framework for machine unlearning based on the following dimensions.
  arXiv  Detail & Related papers  (2024-10-31T20:54:35Z)
• Activate and Reject: Towards Safe Domain Generalization under Category Shift [71.95548187205736]
  We study a practical problem of Domain Generalization under Category Shift (DGCS) It aims to simultaneously detect unknown-class samples and classify known-class samples in the target domains. Compared to prior DG works, we face two new challenges: 1) how to learn the concept of unknown'' during training with only source known-class samples, and 2) how to adapt the source-trained model to unseen environments.
  arXiv  Detail & Related papers  (2023-10-07T07:53:12Z)
• MultIOD: Rehearsal-free Multihead Incremental Object Detector [17.236182938227163]
  We propose MultIOD, a class-incremental object detector based on CenterNet. We employ transfer learning between classes learned initially and those learned incrementally to tackle catastrophic forgetting. Results show that our method outperforms state-of-the-art methods on two Pascal VOC datasets.
  arXiv  Detail & Related papers  (2023-09-11T09:32:45Z)
• Generative Forests [23.554594285885273]
  We focus on generative AI for a type of data that still represent one of the most prevalent form of data: tabular data. Our paper introduces a new powerful class of forest-based models fit for such tasks and a simple training algorithm with strong convergence guarantees. Additional experiments on these tasks reveal that our models can be notably good contenders to diverse state of the art methods.
  arXiv  Detail & Related papers  (2023-08-07T14:58:53Z)
• Neural Collapse Terminus: A Unified Solution for Class Incremental Learning and Its Variants [166.916517335816]
  In this paper, we offer a unified solution to the misalignment dilemma in the three tasks. We propose neural collapse terminus that is a fixed structure with the maximal equiangular inter-class separation for the whole label space. Our method holds the neural collapse optimality in an incremental fashion regardless of data imbalance or data scarcity.
  arXiv  Detail & Related papers  (2023-08-03T13:09:59Z)
• Complementary Learning Subnetworks for Parameter-Efficient Class-Incremental Learning [40.13416912075668]
  We propose a rehearsal-free CIL approach that learns continually via the synergy between two Complementary Learning Subnetworks. Our method achieves competitive results against state-of-the-art methods, especially in accuracy gain, memory cost, training efficiency, and task-order.
  arXiv  Detail & Related papers  (2023-06-21T01:43:25Z)
• TWINS: A Fine-Tuning Framework for Improved Transferability of Adversarial Robustness and Generalization [89.54947228958494]
  This paper focuses on the fine-tuning of an adversarially pre-trained model in various classification tasks. We propose a novel statistics-based approach, Two-WIng NormliSation (TWINS) fine-tuning framework. TWINS is shown to be effective on a wide range of image classification datasets in terms of both generalization and robustness.
  arXiv  Detail & Related papers  (2023-03-20T14:12:55Z)
• Prototypical quadruplet for few-shot class incremental learning [24.814045065163135]
  We propose a novel method that improves classification robustness by identifying a better embedding space using an improved contrasting loss. Our approach retains previously acquired knowledge in the embedding space, even when trained with new classes. We demonstrate the effectiveness of our method by showing that the embedding space remains intact after training the model with new classes and outperforms existing state-of-the-art algorithms in terms of accuracy across different sessions.
  arXiv  Detail & Related papers  (2022-11-05T17:19:14Z)
• Novel Class Discovery without Forgetting [72.52222295216062]
  We identify and formulate a new, pragmatic problem setting of NCDwF: Novel Class Discovery without Forgetting. We propose a machine learning model to incrementally discover novel categories of instances from unlabeled data. We introduce experimental protocols based on CIFAR-10, CIFAR-100 and ImageNet-1000 to measure the trade-off between knowledge retention and novel class discovery.
  arXiv  Detail & Related papers  (2022-07-21T17:54:36Z)
• Rethinking Bayesian Learning for Data Analysis: The Art of Prior and Inference in Sparsity-Aware Modeling [20.296566563098057]
  Sparse modeling for signal processing and machine learning has been at the focus of scientific research for over two decades. This article reviews some recent advances in incorporating sparsity-promoting priors into three popular data modeling tools.
  arXiv  Detail & Related papers  (2022-05-28T00:43:52Z)
• Continual Learning with Bayesian Model based on a Fixed Pre-trained Feature Extractor [55.9023096444383]
  Current deep learning models are characterised by catastrophic forgetting of old knowledge when learning new classes. Inspired by the process of learning new knowledge in human brains, we propose a Bayesian generative model for continual learning.
  arXiv  Detail & Related papers  (2022-04-28T08:41:51Z)
• Simple Stochastic and Online Gradient DescentAlgorithms for Pairwise Learning [65.54757265434465]
  Pairwise learning refers to learning tasks where the loss function depends on a pair instances. Online descent (OGD) is a popular approach to handle streaming data in pairwise learning. In this paper, we propose simple and online descent to methods for pairwise learning.
  arXiv  Detail & Related papers  (2021-11-23T18:10:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.