Learning Prompt with Distribution-Based Feature Replay for Few-Shot Class-Incremental Learning

• URL: http://arxiv.org/abs/2401.01598v2
• Date: Fri, 5 Apr 2024 08:23:29 GMT
• Title: Learning Prompt with Distribution-Based Feature Replay for Few-Shot Class-Incremental Learning
• Authors: Zitong Huang, Ze Chen, Zhixing Chen, Erjin Zhou, Xinxing Xu, Rick Siow Mong Goh, Yong Liu, Wangmeng Zuo, Chunmei Feng,
• Abstract summary: We propose a simple yet effective framework, named Learning Prompt with Distribution-based Feature Replay (LP-DiF) To prevent the learnable prompt from forgetting old knowledge in the new session, we propose a pseudo-feature replay approach. When progressing to a new session, pseudo-features are sampled from old-class distributions combined with training images of the current session to optimize the prompt.
• Score: 56.29097276129473
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Few-shot Class-Incremental Learning (FSCIL) aims to continuously learn new classes based on very limited training data without forgetting the old ones encountered. Existing studies solely relied on pure visual networks, while in this paper we solved FSCIL by leveraging the Vision-Language model (e.g., CLIP) and propose a simple yet effective framework, named Learning Prompt with Distribution-based Feature Replay (LP-DiF). We observe that simply using CLIP for zero-shot evaluation can substantially outperform the most influential methods. Then, prompt tuning technique is involved to further improve its adaptation ability, allowing the model to continually capture specific knowledge from each session. To prevent the learnable prompt from forgetting old knowledge in the new session, we propose a pseudo-feature replay approach. Specifically, we preserve the old knowledge of each class by maintaining a feature-level Gaussian distribution with a diagonal covariance matrix, which is estimated by the image features of training images and synthesized features generated from a VAE. When progressing to a new session, pseudo-features are sampled from old-class distributions combined with training images of the current session to optimize the prompt, thus enabling the model to learn new knowledge while retaining old knowledge. Experiments on three prevalent benchmarks, i.e., CIFAR100, mini-ImageNet, CUB-200, and two more challenging benchmarks, i.e., SUN-397 and CUB-200$^*$ proposed in this paper showcase the superiority of LP-DiF, achieving new state-of-the-art (SOTA) in FSCIL. Code is publicly available at https://github.com/1170300714/LP-DiF.

Related papers

• Knowledge Adaptation Network for Few-Shot Class-Incremental Learning [23.90555521006653]
  Few-shot class-incremental learning aims to incrementally recognize new classes using a few samples. One of the effective methods to solve this challenge is to construct prototypical evolution classifiers. Because representations for new classes are weak and biased, we argue such a strategy is suboptimal.
  arXiv  Detail & Related papers  (2024-09-18T07:51:38Z)
• CEAT: Continual Expansion and Absorption Transformer for Non-Exemplar Class-Incremental Learning [34.59310641291726]
  In real-world applications, dynamic scenarios require the models to possess the capability to learn new tasks continuously without forgetting the old knowledge. We propose a new architecture, named continual expansion and absorption transformer(CEAT) The model can learn the novel knowledge by extending the expanded-fusion layers in parallel with the frozen previous parameters. To improve the learning ability of the model, we designed a novel prototype contrastive loss to reduce the overlap between old and new classes in the feature space.
  arXiv  Detail & Related papers  (2024-03-11T12:40:12Z)
• Exploring Effective Factors for Improving Visual In-Context Learning [56.14208975380607]
  In-Context Learning (ICL) is to understand a new task via a few demonstrations (aka. prompt) and predict new inputs without tuning the models. This paper shows that prompt selection and prompt fusion are two major factors that have a direct impact on the inference performance of visual context learning. We propose a simple framework prompt-SelF for visual in-context learning.
  arXiv  Detail & Related papers  (2023-04-10T17:59:04Z)
• SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Model [73.80068155830708]
  We present an extensive analysis for continual learning on a pre-trained model (CLPM) We propose a simple but extremely effective approach named Slow Learner with Alignment (SLCA) Across a variety of scenarios, our proposal provides substantial improvements for CLPM.
  arXiv  Detail & Related papers  (2023-03-09T08:57:01Z)
• A Closer Look at Few-Shot Video Classification: A New Baseline and Benchmark [33.86872697028233]
  We present an in-depth study on few-shot video classification by making three contributions. First, we perform a consistent comparative study on the existing metric-based methods to figure out their limitations in representation learning. Second, we discover that there is a high correlation between the novel action class and the ImageNet object class, which is problematic in the few-shot recognition setting. Third, we present a new benchmark with more base data to facilitate future few-shot video classification without pre-training.
  arXiv  Detail & Related papers  (2021-10-24T06:01:46Z)
• Few Shot Activity Recognition Using Variational Inference [9.371378627575883]
  We propose a novel variational inference based architectural framework (HF-AR) for few shot activity recognition. Our framework leverages volume-preserving Householder Flow to learn a flexible posterior distribution of the novel classes. This results in better performance as compared to state-of-the-art few shot approaches for human activity recognition.
  arXiv  Detail & Related papers  (2021-08-20T03:57:58Z)
• Few-Shot Incremental Learning with Continually Evolved Classifiers [46.278573301326276]
  Few-shot class-incremental learning (FSCIL) aims to design machine learning algorithms that can continually learn new concepts from a few data points. The difficulty lies in that limited data from new classes not only lead to significant overfitting issues but also exacerbate the notorious catastrophic forgetting problems. We propose a Continually Evolved CIF ( CEC) that employs a graph model to propagate context information between classifiers for adaptation.
  arXiv  Detail & Related papers  (2021-04-07T10:54:51Z)
• Improving Calibration for Long-Tailed Recognition [68.32848696795519]
  We propose two methods to improve calibration and performance in such scenarios. For dataset bias due to different samplers, we propose shifted batch normalization. Our proposed methods set new records on multiple popular long-tailed recognition benchmark datasets.
  arXiv  Detail & Related papers  (2021-04-01T13:55:21Z)
• Few-Shot Class-Incremental Learning [68.75462849428196]
  We focus on a challenging but practical few-shot class-incremental learning (FSCIL) problem. FSCIL requires CNN models to incrementally learn new classes from very few labelled samples, without forgetting the previously learned ones. We represent the knowledge using a neural gas (NG) network, which can learn and preserve the topology of the feature manifold formed by different classes.
  arXiv  Detail & Related papers  (2020-04-23T03:38:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.