Bypassing Logits Bias in Online Class-Incremental Learning with a Generative Framework

• URL: http://arxiv.org/abs/2205.09347v1
• Date: Thu, 19 May 2022 06:54:20 GMT
• Title: Bypassing Logits Bias in Online Class-Incremental Learning with a Generative Framework
• Authors: Gehui Shen, Shibo Jie, Ziheng Li, Zhi-Hong Deng
• Abstract summary: We focus on online class-incremental learning setting in which new classes emerge over time. Almost all existing methods are replay-based with a softmax classifier. We propose a novel generative framework based on the feature space.
• Score: 15.345043222622158
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Continual learning requires the model to maintain the learned knowledge while learning from a non-i.i.d data stream continually. Due to the single-pass training setting, online continual learning is very challenging, but it is closer to the real-world scenarios where quick adaptation to new data is appealing. In this paper, we focus on online class-incremental learning setting in which new classes emerge over time. Almost all existing methods are replay-based with a softmax classifier. However, the inherent logits bias problem in the softmax classifier is a main cause of catastrophic forgetting while existing solutions are not applicable for online settings. To bypass this problem, we abandon the softmax classifier and propose a novel generative framework based on the feature space. In our framework, a generative classifier which utilizes replay memory is used for inference, and the training objective is a pair-based metric learning loss which is proven theoretically to optimize the feature space in a generative way. In order to improve the ability to learn new data, we further propose a hybrid of generative and discriminative loss to train the model. Extensive experiments on several benchmarks, including newly introduced task-free datasets, show that our method beats a series of state-of-the-art replay-based methods with discriminative classifiers, and reduces catastrophic forgetting consistently with a remarkable margin.

Related papers

• Random Representations Outperform Online Continually Learned Representations [68.42776779425978]
  We show that existing online continually trained deep networks produce inferior representations compared to a simple pre-defined random transforms. Our method, called RanDumb, significantly outperforms state-of-the-art continually learned representations across all online continual learning benchmarks. Our study reveals the significant limitations of representation learning, particularly in low-exemplar and online continual learning scenarios.
  arXiv  Detail & Related papers  (2024-02-13T22:07:29Z)
• Enhancing Consistency and Mitigating Bias: A Data Replay Approach for Incremental Learning [100.7407460674153]
  Deep learning systems are prone to catastrophic forgetting when learning from a sequence of tasks. To mitigate the problem, a line of methods propose to replay the data of experienced tasks when learning new tasks. However, it is not expected in practice considering the memory constraint or data privacy issue. As a replacement, data-free data replay methods are proposed by inverting samples from the classification model.
  arXiv  Detail & Related papers  (2024-01-12T12:51:12Z)
• Few-Shot Class-Incremental Learning via Training-Free Prototype Calibration [67.69532794049445]
  We find a tendency for existing methods to misclassify the samples of new classes into base classes, which leads to the poor performance of new classes. We propose a simple yet effective Training-frEE calibratioN (TEEN) strategy to enhance the discriminability of new classes.
  arXiv  Detail & Related papers  (2023-12-08T18:24:08Z)
• Towards Robust Continual Learning with Bayesian Adaptive Moment Regularization [51.34904967046097]
  Continual learning seeks to overcome the challenge of catastrophic forgetting, where a model forgets previously learnt information. We introduce a novel prior-based method that better constrains parameter growth, reducing catastrophic forgetting. Results show that BAdam achieves state-of-the-art performance for prior-based methods on challenging single-headed class-incremental experiments.
  arXiv  Detail & Related papers  (2023-09-15T17:10:51Z)
• Complementary Learning Subnetworks for Parameter-Efficient Class-Incremental Learning [40.13416912075668]
  We propose a rehearsal-free CIL approach that learns continually via the synergy between two Complementary Learning Subnetworks. Our method achieves competitive results against state-of-the-art methods, especially in accuracy gain, memory cost, training efficiency, and task-order.
  arXiv  Detail & Related papers  (2023-06-21T01:43:25Z)
• Class-Incremental Learning: A Survey [84.30083092434938]
  Class-Incremental Learning (CIL) enables the learner to incorporate the knowledge of new classes incrementally. CIL tends to catastrophically forget the characteristics of former ones, and its performance drastically degrades. We provide a rigorous and unified evaluation of 17 methods in benchmark image classification tasks to find out the characteristics of different algorithms.
  arXiv  Detail & Related papers  (2023-02-07T17:59:05Z)
• Prototypical quadruplet for few-shot class incremental learning [24.814045065163135]
  We propose a novel method that improves classification robustness by identifying a better embedding space using an improved contrasting loss. Our approach retains previously acquired knowledge in the embedding space, even when trained with new classes. We demonstrate the effectiveness of our method by showing that the embedding space remains intact after training the model with new classes and outperforms existing state-of-the-art algorithms in terms of accuracy across different sessions.
  arXiv  Detail & Related papers  (2022-11-05T17:19:14Z)
• Learning to Generate Novel Classes for Deep Metric Learning [24.048915378172012]
  We introduce a new data augmentation approach that synthesizes novel classes and their embedding vectors. We implement this idea by learning and exploiting a conditional generative model, which, given a class label and a noise, produces a random embedding vector of the class. Our proposed generator allows the loss to use richer class relations by augmenting realistic and diverse classes, resulting in better generalization to unseen samples.
  arXiv  Detail & Related papers  (2022-01-04T06:55:19Z)
• Class-Incremental Learning with Generative Classifiers [6.570917734205559]
  We propose a new strategy for class-incremental learning: generative classification. Our proposal is to learn the joint distribution p(x,y), factorized as p(x|y)p(y), and to perform classification using Bayes' rule. As a proof-of-principle, here we implement this strategy by training a variational autoencoder for each class to be learned.
  arXiv  Detail & Related papers  (2021-04-20T16:26:14Z)
• Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learning [17.310385256678654]
  Class-incremental continual learning (CL) studies the problem of learning new classes continually from an online non-stationary data stream. While memory replay has shown promising results, the recency bias in online learning caused by the commonly used Softmax classifier remains an unsolved challenge. Although the Nearest-Class-Mean (NCM) classifier is significantly undervalued in the CL community, we demonstrate that it is a simple yet effective substitute for the Softmax classifier.
  arXiv  Detail & Related papers  (2021-03-22T20:27:34Z)
• Learning Adaptive Embedding Considering Incremental Class [55.21855842960139]
  Class-Incremental Learning (CIL) aims to train a reliable model with the streaming data, which emerges unknown classes sequentially. Different from traditional closed set learning, CIL has two main challenges: 1) Novel class detection. After the novel classes are detected, the model needs to be updated without re-training using entire previous data.
  arXiv  Detail & Related papers  (2020-08-31T04:11:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.