Task Confusion and Catastrophic Forgetting in Class-Incremental Learning: A Mathematical Framework for Discriminative and Generative Modelings

• URL: http://arxiv.org/abs/2410.20768v1
• Date: Mon, 28 Oct 2024 06:08:38 GMT
• Title: Task Confusion and Catastrophic Forgetting in Class-Incremental Learning: A Mathematical Framework for Discriminative and Generative Modelings
• Authors: Milad Khademi Nori, Il-Min Kim,
• Abstract summary: In class-incremental learning (class-IL), models must classify all previously seen classes at test time without task-IDs, leading to task confusion. We present a novel mathematical framework for class-IL and prove the Infeasibility Theorem, showing optimal class-IL is impossible with discriminative modeling.
• Score: 5.899701834228992
• License:
• Abstract: In class-incremental learning (class-IL), models must classify all previously seen classes at test time without task-IDs, leading to task confusion. Despite being a key challenge, task confusion lacks a theoretical understanding. We present a novel mathematical framework for class-IL and prove the Infeasibility Theorem, showing optimal class-IL is impossible with discriminative modeling due to task confusion. However, we establish the Feasibility Theorem, demonstrating that generative modeling can achieve optimal class-IL by overcoming task confusion. We then assess popular class-IL strategies, including regularization, bias-correction, replay, and generative classifier, using our framework. Our analysis suggests that adopting generative modeling, either for generative replay or direct classification (generative classifier), is essential for optimal class-IL.

Related papers

• Generative Multi-modal Models are Good Class-Incremental Learners [51.5648732517187]
  We propose a novel generative multi-modal model (GMM) framework for class-incremental learning. Our approach directly generates labels for images using an adapted generative model. Under the Few-shot CIL setting, we have improved by at least 14% accuracy over all the current state-of-the-art methods with significantly less forgetting.
  arXiv  Detail & Related papers  (2024-03-27T09:21:07Z)
• Learn to Categorize or Categorize to Learn? Self-Coding for Generalized Category Discovery [49.1865089933055]
  We propose a novel, efficient and self-supervised method capable of discovering previously unknown categories at test time. A salient feature of our approach is the assignment of minimum length category codes to individual data instances. Experimental evaluations, bolstered by state-of-the-art benchmark comparisons, testify to the efficacy of our solution.
  arXiv  Detail & Related papers  (2023-10-30T17:45:32Z)
• Learning Optimal Fair Scoring Systems for Multi-Class Classification [0.0]
  There are growing concerns about Machine Learning models with respect to their lack of interpretability and the undesirable biases they can generate or reproduce. In this paper, we use Mixed-Integer Linear Programming (MILP) techniques to produce inherently interpretable scoring systems under sparsity and fairness constraints.
  arXiv  Detail & Related papers  (2023-04-11T07:18:04Z)
• Generalization Bounds for Few-Shot Transfer Learning with Pretrained Classifiers [26.844410679685424]
  We study the ability of foundation models to learn representations for classification that are transferable to new, unseen classes. We show that the few-shot error of the learned feature map on new classes is small in case of class-feature-variability collapse.
  arXiv  Detail & Related papers  (2022-12-23T18:46:05Z)
• Anomaly Detection using Ensemble Classification and Evidence Theory [62.997667081978825]
  We present a novel approach for novel detection using ensemble classification and evidence theory. A pool selection strategy is presented to build a solid ensemble classifier. We use uncertainty for the anomaly detection approach.
  arXiv  Detail & Related papers  (2022-12-23T00:50:41Z)
• Evidential Deep Learning for Class-Incremental Semantic Segmentation [15.563703446465823]
  Class-Incremental Learning is a challenging problem in machine learning that aims to extend previously trained neural networks with new classes. In this paper, we address the problem of how to model unlabeled classes while avoiding spurious feature clustering of future uncorrelated classes. Our method factorizes the problem into a separate foreground class probability, calculated by the expected value of the Dirichlet distribution, and an unknown class (background) probability corresponding to the uncertainty of the estimate.
  arXiv  Detail & Related papers  (2022-12-06T10:13:30Z)
• Parametric Classification for Generalized Category Discovery: A Baseline Study [70.73212959385387]
  Generalized Category Discovery (GCD) aims to discover novel categories in unlabelled datasets using knowledge learned from labelled samples. We investigate the failure of parametric classifiers, verify the effectiveness of previous design choices when high-quality supervision is available, and identify unreliable pseudo-labels as a key problem. We propose a simple yet effective parametric classification method that benefits from entropy regularisation, achieves state-of-the-art performance on multiple GCD benchmarks and shows strong robustness to unknown class numbers.
  arXiv  Detail & Related papers  (2022-11-21T18:47:11Z)
• CLASTER: Clustering with Reinforcement Learning for Zero-Shot Action Recognition [52.66360172784038]
  We propose a clustering-based model, which considers all training samples at once, instead of optimizing for each instance individually. We call the proposed method CLASTER and observe that it consistently improves over the state-of-the-art in all standard datasets.
  arXiv  Detail & Related papers  (2021-01-18T12:46:24Z)
• Learning and Evaluating Representations for Deep One-class Classification [59.095144932794646]
  We present a two-stage framework for deep one-class classification. We first learn self-supervised representations from one-class data, and then build one-class classifiers on learned representations. In experiments, we demonstrate state-of-the-art performance on visual domain one-class classification benchmarks.
  arXiv  Detail & Related papers  (2020-11-04T23:33:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.