Few-shot Tuning of Foundation Models for Class-incremental Learning

• URL: http://arxiv.org/abs/2405.16625v1
• Date: Sun, 26 May 2024 16:41:03 GMT
• Title: Few-shot Tuning of Foundation Models for Class-incremental Learning
• Authors: Shuvendu Roy, Elham Dolatabadi, Arash Afkanpour, Ali Etemad,
• Abstract summary: We propose a new approach to continually tune foundation models for new classes in few-shot settings. CoACT shows up to 13.5% improvement in standard FSCIL over the current SOTA on benchmark evaluations.
• Score: 19.165004570789755
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: For the first time, we explore few-shot tuning of vision foundation models for class-incremental learning. Unlike existing few-shot class incremental learning (FSCIL) methods, which train an encoder on a base session to ensure forward compatibility for future continual learning, foundation models are generally trained on large unlabelled data without such considerations. This renders prior methods from traditional FSCIL incompatible for FSCIL with the foundation model. To this end, we propose Consistency-guided Asynchronous Contrastive Tuning (CoACT), a new approach to continually tune foundation models for new classes in few-shot settings. CoACT comprises three components: (i) asynchronous contrastive tuning, which learns new classes by including LoRA modules in the pre-trained encoder, while enforcing consistency between two asynchronous encoders; (ii) controlled fine-tuning, which facilitates effective tuning of a subset of the foundation model; and (iii) consistency-guided incremental tuning, which enforces additional regularization during later sessions to reduce forgetting of the learned classes. We perform an extensive study on 16 diverse datasets and demonstrate the effectiveness of CoACT, outperforming the best baseline method by 2.47% on average and with up to 12.52% on individual datasets. Additionally, CoACT shows reduced forgetting and robustness in low-shot experiments. As an added bonus, CoACT shows up to 13.5% improvement in standard FSCIL over the current SOTA on benchmark evaluations. We make our code publicly available at https://github.com/ShuvenduRoy/CoACT-FSCIL.

Related papers

• DATA: Decomposed Attention-based Task Adaptation for Rehearsal-Free Continual Learning [22.386864304549285]
  Continual learning (CL) is essential for Large Language Models (LLMs) to adapt to evolving real-world demands. Recent rehearsal-free methods employ model-based and regularization-based strategies to address this issue. We propose a $textbfD$e $textbfA$ttention-based $textbfTask $textbfA$daptation ( DATA) DATA explicitly decouples and learns both task-specific and task-shared knowledge using high-rank and low-rank task adapters.
  arXiv  Detail & Related papers  (2025-02-17T06:35:42Z)
• Towards Compatible Fine-tuning for Vision-Language Model Updates [114.25776195225494]
  Class-conditioned Context Optimization (ContCoOp) integrates learnable prompts with class embeddings using an attention layer before inputting them into the text encoder. Our experiments over 15 datasets show that our ContCoOp achieves the highest compatibility over the baseline methods, and exhibits robust out-of-distribution generalization.
  arXiv  Detail & Related papers  (2024-12-30T12:06:27Z)
• High-Performance Few-Shot Segmentation with Foundation Models: An Empirical Study [64.06777376676513]
  We develop a few-shot segmentation (FSS) framework based on foundation models. To be specific, we propose a simple approach to extract implicit knowledge from foundation models to construct coarse correspondence. Experiments on two widely used datasets demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2024-09-10T08:04:11Z)
• Combining Denoising Autoencoders with Contrastive Learning to fine-tune Transformer Models [0.0]
  This work proposes a 3 Phase technique to adjust a base model for a classification task. We adapt the model's signal to the data distribution by performing further training with a Denoising Autoencoder (DAE) In addition, we introduce a new data augmentation approach for Supervised Contrastive Learning to correct the unbalanced datasets.
  arXiv  Detail & Related papers  (2024-05-23T11:08:35Z)
• Rethinking Few-shot 3D Point Cloud Semantic Segmentation [62.80639841429669]
  This paper revisits few-shot 3D point cloud semantic segmentation (FS-PCS) We focus on two significant issues in the state-of-the-art: foreground leakage and sparse point distribution. To address these issues, we introduce a standardized FS-PCS setting, upon which a new benchmark is built.
  arXiv  Detail & Related papers  (2024-03-01T15:14:47Z)
• Generalized Logit Adjustment: Calibrating Fine-tuned Models by Removing Label Bias in Foundation Models [75.9543301303586]
  Foundation models like CLIP allow zero-shot transfer on various tasks without additional training data. Fine-tuning and ensembling are also commonly adopted to better fit the downstream tasks. However, we argue that prior work has overlooked the inherent biases in foundation models.
  arXiv  Detail & Related papers  (2023-10-12T08:01:11Z)
• Universal Domain Adaptation from Foundation Models: A Baseline Study [58.51162198585434]
  We make empirical studies of state-of-the-art UniDA methods using foundation models. We introduce textitCLIP distillation, a parameter-free method specifically designed to distill target knowledge from CLIP models. Although simple, our method outperforms previous approaches in most benchmark tasks.
  arXiv  Detail & Related papers  (2023-05-18T16:28:29Z)
• TWINS: A Fine-Tuning Framework for Improved Transferability of Adversarial Robustness and Generalization [89.54947228958494]
  This paper focuses on the fine-tuning of an adversarially pre-trained model in various classification tasks. We propose a novel statistics-based approach, Two-WIng NormliSation (TWINS) fine-tuning framework. TWINS is shown to be effective on a wide range of image classification datasets in terms of both generalization and robustness.
  arXiv  Detail & Related papers  (2023-03-20T14:12:55Z)
• Modeling Inter-Class and Intra-Class Constraints in Novel Class Discovery [20.67503042774617]
  Novel class discovery (NCD) aims at learning a model that transfers the common knowledge from a class-disjoint labelled dataset to another unlabelled dataset. We propose to model both inter-class and intra-class constraints in NCD based on the symmetric Kullback-Leibler divergence (sKLD)
  arXiv  Detail & Related papers  (2022-10-07T14:46:32Z)
• Rethinking Few-Shot Class-Incremental Learning with Open-Set Hypothesis in Hyperbolic Geometry [21.38183613466714]
  Few-Shot Class-Incremental Learning (FSCIL) aims at incrementally learning novel classes from a few labeled samples. In this paper, we rethink the configuration of FSCIL with the open-set hypothesis by reserving the possibility in the first session for incoming categories. To assign better performances on both close-set and open-set recognition to the model, Hyperbolic Reciprocal Point Learning module (Hyper-RPL) is built on Reciprocal Point Learning (RPL) with hyperbolic neural networks.
  arXiv  Detail & Related papers  (2022-07-20T15:13:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.