PECTP: Parameter-Efficient Cross-Task Prompts for Incremental Vision Transformer

• URL: http://arxiv.org/abs/2407.03813v1
• Date: Thu, 4 Jul 2024 10:37:58 GMT
• Title: PECTP: Parameter-Efficient Cross-Task Prompts for Incremental Vision Transformer
• Authors: Qian Feng, Hanbin Zhao, Chao Zhang, Jiahua Dong, Henghui Ding, Yu-Gang Jiang, Hui Qian,
• Abstract summary: Incremental Learning (IL) aims to learn deep models on sequential tasks continually. Recent vast pre-trained models (PTMs) have achieved outstanding performance by prompt technique in practical IL without the old samples.
• Score: 76.39111896665585
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Incremental Learning (IL) aims to learn deep models on sequential tasks continually, where each new task includes a batch of new classes and deep models have no access to task-ID information at the inference time. Recent vast pre-trained models (PTMs) have achieved outstanding performance by prompt technique in practical IL without the old samples (rehearsal-free) and with a memory constraint (memory-constrained): Prompt-extending and Prompt-fixed methods. However, prompt-extending methods need a large memory buffer to maintain an ever-expanding prompt pool and meet an extra challenging prompt selection problem. Prompt-fixed methods only learn a single set of prompts on one of the incremental tasks and can not handle all the incremental tasks effectively. To achieve a good balance between the memory cost and the performance on all the tasks, we propose a Parameter-Efficient Cross-Task Prompt (PECTP) framework with Prompt Retention Module (PRM) and classifier Head Retention Module (HRM). To make the final learned prompts effective on all incremental tasks, PRM constrains the evolution of cross-task prompts' parameters from Outer Prompt Granularity and Inner Prompt Granularity. Besides, we employ HRM to inherit old knowledge in the previously learned classifier heads to facilitate the cross-task prompts' generalization ability. Extensive experiments show the effectiveness of our method. The source codes will be available at \url{https://github.com/RAIAN08/PECTP}.

Related papers

• LW2G: Learning Whether to Grow for Prompt-based Continual Learning [15.766350352592331]
  Recent Prompt-based Continual Learning (PCL) has achieved remarkable performance with Pre-Trained Models (PTMs) We propose a plug-in module in the former stage to textbfLearn Whether to Grow (LW2G) based on the disparities between tasks. Inspired by Gradient Projection Continual Learning, our LW2G develops a metric called Hinder Forward Capability (HFC) to measure the hindrance imposed on learning new tasks.
  arXiv  Detail & Related papers  (2024-09-27T15:55:13Z)
• Less is more: Summarizing Patch Tokens for efficient Multi-Label Class-Incremental Learning [38.36863497458095]
  We propose a new class-incremental learning method Multi-Label class incremental learning via summarising pAtch tokeN Embeddings (MULTI-LANE) Our proposed method Multi-Label class incremental learning via summarising pAtch tokeN Embeddings (MULTI-LANE) enables learning disentangled task-specific representations in MLCIL while ensuring fast inference.
  arXiv  Detail & Related papers  (2024-05-24T15:18:27Z)
• Convolutional Prompting meets Language Models for Continual Learning [4.115213208594654]
  Continual Learning (CL) enables machine learning models to learn from continuously shifting new training data in absence of data from old tasks. We propose ConvPrompt, a novel convolutional prompt creation mechanism that maintains layer-wise shared embeddings. The intelligent use of convolution enables us to maintain a low parameter overhead without compromising performance.
  arXiv  Detail & Related papers  (2024-03-29T17:40:37Z)
• Few-Shot Class Incremental Learning with Attention-Aware Self-Adaptive Prompt [58.880105981772324]
  We propose a novel framework named Attention-aware Self-adaptive Prompt (ASP) ASP encourages task-invariant prompts to capture shared knowledge by reducing specific information from the attention aspect. In summary, ASP prevents overfitting on base task and does not require enormous data in few-shot incremental tasks.
  arXiv  Detail & Related papers  (2024-03-14T20:34:53Z)
• Introducing Language Guidance in Prompt-based Continual Learning [95.03110230754423]
  We propose Language Guidance for Prompt-based Continual Learning (LGCL) as a plug-in for prompt-based methods. LGCL consistently improves the performance of prompt-based continual learning methods to set a new state-of-the art.
  arXiv  Detail & Related papers  (2023-08-30T08:03:49Z)
• TransPrompt v2: A Transferable Prompting Framework for Cross-task Text Classification [37.824031151922604]
  We propose TransPrompt v2, a novel transferable prompting framework for few-shot learning across similar or distant text classification tasks. For learning across similar tasks, we employ a multi-task meta-knowledge acquisition (MMA) procedure to train a meta-learner. For learning across distant tasks, we inject the task type descriptions into the prompt, and capture the intra-type and inter-type prompt embeddings.
  arXiv  Detail & Related papers  (2023-08-29T04:16:57Z)
• Decomposed Prompting: A Modular Approach for Solving Complex Tasks [55.42850359286304]
  We propose Decomposed Prompting to solve complex tasks by decomposing them (via prompting) into simpler sub-tasks. This modular structure allows each prompt to be optimized for its specific sub-task. We show that the flexibility and modularity of Decomposed Prompting allows it to outperform prior work on few-shot prompting.
  arXiv  Detail & Related papers  (2022-10-05T17:28:20Z)
• Instance-wise Prompt Tuning for Pretrained Language Models [72.74916121511662]
  Instance-wise Prompt Tuning (IPT) is the first prompt learning paradigm that injects knowledge from the input data instances to the prompts. IPT significantly outperforms task-based prompt learning methods, and achieves comparable performance to conventional finetuning with only 0.5% - 1.5% of tuned parameters.
  arXiv  Detail & Related papers  (2022-06-04T10:08:50Z)
• Continual Prompt Tuning for Dialog State Tracking [58.66412648276873]
  A desirable dialog system should be able to continually learn new skills without forgetting old ones. We present Continual Prompt Tuning, a parameter-efficient framework that not only avoids forgetting but also enables knowledge transfer between tasks.
  arXiv  Detail & Related papers  (2022-03-13T13:22:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.