Multimodal Parameter-Efficient Few-Shot Class Incremental Learning
- URL: http://arxiv.org/abs/2303.04751v2
- Date: Mon, 8 Jan 2024 12:28:19 GMT
- Title: Multimodal Parameter-Efficient Few-Shot Class Incremental Learning
- Authors: Marco D'Alessandro, Alberto Alonso, Enrique Calabr\'es, Mikel Galar
- Abstract summary: Few-Shot Class Incremental Learning (FSCIL) is a challenging continual learning task, where limited training examples are available during several learning sessions.
To succeed in this task, it is necessary to avoid over-fitting new classes caused by biased distributions in the few-shot training sets.
CPE-CLIP significantly improves FSCIL performance compared to state-of-the-art proposals while also drastically reducing the number of learnable parameters and training costs.
- Score: 1.9220716793379256
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Few-Shot Class Incremental Learning (FSCIL) is a challenging continual
learning task, where limited training examples are available during several
learning sessions. To succeed in this task, it is necessary to avoid
over-fitting new classes caused by biased distributions in the few-shot
training sets. The general approach to address this issue involves enhancing
the representational capability of a pre-defined backbone architecture by
adding special modules for backward compatibility with older classes. However,
this approach has not yet solved the dilemma of ensuring high classification
accuracy over time while reducing the gap between the performance obtained on
larger training sets and the smaller ones. In this work, we propose an
alternative approach called Continual Parameter-Efficient CLIP (CPE-CLIP) to
reduce the loss of information between different learning sessions. Instead of
adapting additional modules to address information loss, we leverage the vast
knowledge acquired by CLIP in large-scale pre-training and its effectiveness in
generalizing to new concepts. Our approach is multimodal and
parameter-efficient, relying on learnable prompts for both the language and
vision encoders to enable transfer learning across sessions. We also introduce
prompt regularization to improve performance and prevent forgetting. Our
experimental results demonstrate that CPE-CLIP significantly improves FSCIL
performance compared to state-of-the-art proposals while also drastically
reducing the number of learnable parameters and training costs.
Related papers
- Beyond Prompt Learning: Continual Adapter for Efficient Rehearsal-Free Continual Learning [22.13331870720021]
We propose a beyond prompt learning approach to the RFCL task, called Continual Adapter (C-ADA)
C-ADA flexibly extends specific weights in CAL to learn new knowledge for each task and freezes old weights to preserve prior knowledge.
Our approach achieves significantly improved performance and training speed, outperforming the current state-of-the-art (SOTA) method.
arXiv Detail & Related papers (2024-07-14T17:40:40Z) - SHERL: Synthesizing High Accuracy and Efficient Memory for Resource-Limited Transfer Learning [63.93193829913252]
We propose an innovative METL strategy called SHERL for resource-limited scenarios.
In the early route, intermediate outputs are consolidated via an anti-redundancy operation.
In the late route, utilizing minimal late pre-trained layers could alleviate the peak demand on memory overhead.
arXiv Detail & Related papers (2024-07-10T10:22:35Z) - Multi-Epoch learning with Data Augmentation for Deep Click-Through Rate Prediction [53.88231294380083]
We introduce a novel Multi-Epoch learning with Data Augmentation (MEDA) framework, suitable for both non-continual and continual learning scenarios.
MEDA minimizes overfitting by reducing the dependency of the embedding layer on subsequent training data.
Our findings confirm that pre-trained layers can adapt to new embedding spaces, enhancing performance without overfitting.
arXiv Detail & Related papers (2024-06-27T04:00:15Z) - Convolutional Prompting meets Language Models for Continual Learning [4.115213208594654]
Continual Learning (CL) enables machine learning models to learn from continuously shifting new training data in absence of data from old tasks.
We propose ConvPrompt, a novel convolutional prompt creation mechanism that maintains layer-wise shared embeddings.
The intelligent use of convolution enables us to maintain a low parameter overhead without compromising performance.
arXiv Detail & Related papers (2024-03-29T17:40:37Z) - Enhanced Few-Shot Class-Incremental Learning via Ensemble Models [34.84881941101568]
Few-shot class-incremental learning aims to continually fit new classes with limited training data.
The main challenges are overfitting the rare new training samples and forgetting old classes.
We propose a new ensemble model framework cooperating with data augmentation to boost generalization.
arXiv Detail & Related papers (2024-01-14T06:07:07Z) - Enhancing Few-shot CLIP with Semantic-Aware Fine-Tuning [61.902254546858465]
Methods based on Contrastive Language-Image Pre-training have exhibited promising performance in few-shot adaptation tasks.
We propose fine-tuning the parameters of the attention pooling layer during the training process to encourage the model to focus on task-specific semantics.
arXiv Detail & Related papers (2023-11-08T05:18:57Z) - Unlearn What You Want to Forget: Efficient Unlearning for LLMs [92.51670143929056]
Large language models (LLMs) have achieved significant progress from pre-training on and memorizing a wide range of textual data.
This process might suffer from privacy issues and violations of data protection regulations.
We propose an efficient unlearning framework that could efficiently update LLMs without having to retrain the whole model after data removals.
arXiv Detail & Related papers (2023-10-31T03:35:59Z) - Complementary Learning Subnetworks for Parameter-Efficient
Class-Incremental Learning [40.13416912075668]
We propose a rehearsal-free CIL approach that learns continually via the synergy between two Complementary Learning Subnetworks.
Our method achieves competitive results against state-of-the-art methods, especially in accuracy gain, memory cost, training efficiency, and task-order.
arXiv Detail & Related papers (2023-06-21T01:43:25Z) - OverPrompt: Enhancing ChatGPT through Efficient In-Context Learning [49.38867353135258]
We propose OverPrompt, leveraging the in-context learning capability of LLMs to handle multiple task inputs.
Our experiments show that OverPrompt can achieve cost-efficient zero-shot classification without causing significant detriment to task performance.
arXiv Detail & Related papers (2023-05-24T10:08:04Z) - Mitigating Forgetting in Online Continual Learning via Contrasting
Semantically Distinct Augmentations [22.289830907729705]
Online continual learning (OCL) aims to enable model learning from a non-stationary data stream to continuously acquire new knowledge as well as retain the learnt one.
Main challenge comes from the "catastrophic forgetting" issue -- the inability to well remember the learnt knowledge while learning the new ones.
arXiv Detail & Related papers (2022-11-10T05:29:43Z) - Efficient Feature Transformations for Discriminative and Generative
Continual Learning [98.10425163678082]
We propose a simple task-specific feature map transformation strategy for continual learning.
Theses provide powerful flexibility for learning new tasks, achieved with minimal parameters added to the base architecture.
We demonstrate the efficacy and efficiency of our method with an extensive set of experiments in discriminative (CIFAR-100 and ImageNet-1K) and generative sequences of tasks.
arXiv Detail & Related papers (2021-03-25T01:48:14Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.