Teamwork Is Not Always Good: An Empirical Study of Classifier Drift in Class-incremental Information Extraction

• URL: http://arxiv.org/abs/2305.16559v1
• Date: Fri, 26 May 2023 00:57:43 GMT
• Title: Teamwork Is Not Always Good: An Empirical Study of Classifier Drift in Class-incremental Information Extraction
• Authors: Minqian Liu, Lifu Huang
• Abstract summary: Class-incremental learning aims to develop a learning system that can continually learn new classes from a data stream without forgetting previously learned classes. In this paper, we take a closer look at how the drift in the classifier leads to forgetting, and accordingly, four simple yet (super-) effective solutions to alleviate the drift. Our solutions consistently show significant improvement over the previous state-of-the-art approaches with up to 44.7% absolute F-score gain.
• Score: 12.4259256312658
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Class-incremental learning (CIL) aims to develop a learning system that can continually learn new classes from a data stream without forgetting previously learned classes. When learning classes incrementally, the classifier must be constantly updated to incorporate new classes, and the drift in decision boundary may lead to severe forgetting. This fundamental challenge, however, has not yet been studied extensively, especially in the setting where no samples from old classes are stored for rehearsal. In this paper, we take a closer look at how the drift in the classifier leads to forgetting, and accordingly, design four simple yet (super-) effective solutions to alleviate the classifier drift: an Individual Classifiers with Frozen Feature Extractor (ICE) framework where we individually train a classifier for each learning session, and its three variants ICE-PL, ICE-O, and ICE-PL&O which further take the logits of previously learned classes from old sessions or a constant logit of an Other class as a constraint to the learning of new classifiers. Extensive experiments and analysis on 6 class-incremental information extraction tasks demonstrate that our solutions, especially ICE-O, consistently show significant improvement over the previous state-of-the-art approaches with up to 44.7% absolute F-score gain, providing a strong baseline and insights for future research on class-incremental learning.

Related papers

• Strategic Base Representation Learning via Feature Augmentations for Few-Shot Class Incremental Learning [1.5269945475810085]
  Few-shot class incremental learning implies the model to learn new classes while retaining knowledge of previously learned classes with a small number of training instances. Existing frameworks typically freeze the parameters of the previously learned classes during the incorporation of new classes. We propose a novel feature augmentation driven contrastive learning framework designed to enhance the separation of previously learned classes to accommodate new classes.
  arXiv  Detail & Related papers  (2025-01-16T08:17:32Z)
• Happy: A Debiased Learning Framework for Continual Generalized Category Discovery [54.54153155039062]
  This paper explores the underexplored task of Continual Generalized Category Discovery (C-GCD) C-GCD aims to incrementally discover new classes from unlabeled data while maintaining the ability to recognize previously learned classes. We introduce a debiased learning framework, namely Happy, characterized by Hardness-aware prototype sampling and soft entropy regularization.
  arXiv  Detail & Related papers  (2024-10-09T04:18:51Z)
• CLOSER: Towards Better Representation Learning for Few-Shot Class-Incremental Learning [52.63674911541416]
  Few-shot class-incremental learning (FSCIL) faces several challenges, such as overfitting and forgetting. Our primary focus is representation learning on base classes to tackle the unique challenge of FSCIL. We find that trying to secure the spread of features within a more confined feature space enables the learned representation to strike a better balance between transferability and discriminability.
  arXiv  Detail & Related papers  (2024-10-08T02:23:16Z)
• Read Between the Layers: Leveraging Multi-Layer Representations for Rehearsal-Free Continual Learning with Pre-Trained Models [15.847302755988506]
  We address the Continual Learning problem, wherein a model must learn a sequence of tasks from non-stationary distributions. We propose LayUP, a new prototype-based approach to CL that leverages second-order feature statistics from multiple intermediate layers of a pre-trained network. Our results demonstrate that fully exhausting the representational capacities of pre-trained models in CL goes well beyond their final embeddings.
  arXiv  Detail & Related papers  (2023-12-13T13:11:44Z)
• Few-Shot Class-Incremental Learning via Training-Free Prototype Calibration [67.69532794049445]
  We find a tendency for existing methods to misclassify the samples of new classes into base classes, which leads to the poor performance of new classes. We propose a simple yet effective Training-frEE calibratioN (TEEN) strategy to enhance the discriminability of new classes.
  arXiv  Detail & Related papers  (2023-12-08T18:24:08Z)
• S3C: Self-Supervised Stochastic Classifiers for Few-Shot Class-Incremental Learning [22.243176199188238]
  Few-shot class-incremental learning (FSCIL) aims to learn progressively about new classes with very few labeled samples, without forgetting the knowledge of already learnt classes. FSCIL suffers from two major challenges: (i) over-fitting on the new classes due to limited amount of data, (ii) catastrophically forgetting about the old classes due to unavailability of data from these classes in the incremental stages.
  arXiv  Detail & Related papers  (2023-07-05T12:41:46Z)
• SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Model [73.80068155830708]
  We present an extensive analysis for continual learning on a pre-trained model (CLPM) We propose a simple but extremely effective approach named Slow Learner with Alignment (SLCA) Across a variety of scenarios, our proposal provides substantial improvements for CLPM.
  arXiv  Detail & Related papers  (2023-03-09T08:57:01Z)
• Generalization Bounds for Few-Shot Transfer Learning with Pretrained Classifiers [26.844410679685424]
  We study the ability of foundation models to learn representations for classification that are transferable to new, unseen classes. We show that the few-shot error of the learned feature map on new classes is small in case of class-feature-variability collapse.
  arXiv  Detail & Related papers  (2022-12-23T18:46:05Z)
• Anomaly Detection using Ensemble Classification and Evidence Theory [62.997667081978825]
  We present a novel approach for novel detection using ensemble classification and evidence theory. A pool selection strategy is presented to build a solid ensemble classifier. We use uncertainty for the anomaly detection approach.
  arXiv  Detail & Related papers  (2022-12-23T00:50:41Z)
• Long-tail Recognition via Compositional Knowledge Transfer [60.03764547406601]
  We introduce a novel strategy for long-tail recognition that addresses the tail classes' few-shot problem. Our objective is to transfer knowledge acquired from information-rich common classes to semantically similar, and yet data-hungry, rare classes. Experiments show that our approach can achieve significant performance boosts on rare classes while maintaining robust common class performance.
  arXiv  Detail & Related papers  (2021-12-13T15:48:59Z)
• Few-Shot Incremental Learning with Continually Evolved Classifiers [46.278573301326276]
  Few-shot class-incremental learning (FSCIL) aims to design machine learning algorithms that can continually learn new concepts from a few data points. The difficulty lies in that limited data from new classes not only lead to significant overfitting issues but also exacerbate the notorious catastrophic forgetting problems. We propose a Continually Evolved CIF ( CEC) that employs a graph model to propagate context information between classifiers for adaptation.
  arXiv  Detail & Related papers  (2021-04-07T10:54:51Z)
• Class-incremental Learning with Pre-allocated Fixed Classifiers [20.74548175713497]
  In class-incremental learning, a learning agent faces a stream of data with the goal of learning new classes while not forgetting previous ones. We propose a novel fixed classifier in which a number of pre-allocated output nodes are subject to the classification loss right from the beginning of the learning phase.
  arXiv  Detail & Related papers  (2020-10-16T22:40:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.