POP: Prompt Of Prompts for Continual Learning

• URL: http://arxiv.org/abs/2306.08200v1
• Date: Wed, 14 Jun 2023 02:09:26 GMT
• Title: POP: Prompt Of Prompts for Continual Learning
• Authors: Zhiyuan Hu, Jiancheng Lyu, Dashan Gao, Nuno Vasconcelos
• Abstract summary: Continual learning (CL) aims to mimic the human ability to learn new concepts without catastrophic forgetting. We show that a foundation model equipped with POP learning is able to outperform classic CL methods by a significant margin.
• Score: 59.15888651733645
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Continual learning (CL) has attracted increasing attention in the recent past. It aims to mimic the human ability to learn new concepts without catastrophic forgetting. While existing CL methods accomplish this to some extent, they are still prone to semantic drift of the learned feature space. Foundation models, which are endowed with a robust feature representation, learned from very large datasets, provide an interesting substrate for the solution of the CL problem. Recent work has also shown that they can be adapted to specific tasks by prompt tuning techniques that leave the generality of the representation mostly unscathed. An open question is, however, how to learn both prompts that are task specific and prompts that are global, i.e. capture cross-task information. In this work, we propose the Prompt Of Prompts (POP) model, which addresses this goal by progressively learning a group of task-specified prompts and a group of global prompts, denoted as POP, to integrate information from the former. We show that a foundation model equipped with POP learning is able to outperform classic CL methods by a significant margin. Moreover, as prompt tuning only requires a small set of training samples, POP is able to perform CL in the few-shot setting, while still outperforming competing methods trained on the entire dataset.

Related papers

• CLEO: Continual Learning of Evolving Ontologies [12.18795037817058]
  Continual learning (CL) aims to instill the lifelong learning of humans in intelligent systems. General learning processes are not just limited to learning information, but also refinement of existing information. CLEO is motivated by the need for intelligent systems to adapt to real-world changes over time.
  arXiv  Detail & Related papers  (2024-07-11T11:32:33Z)
• PECTP: Parameter-Efficient Cross-Task Prompts for Incremental Vision Transformer [76.39111896665585]
  Incremental Learning (IL) aims to learn deep models on sequential tasks continually. Recent vast pre-trained models (PTMs) have achieved outstanding performance by prompt technique in practical IL without the old samples.
  arXiv  Detail & Related papers  (2024-07-04T10:37:58Z)
• Instructing Prompt-to-Prompt Generation for Zero-Shot Learning [116.33775552866476]
  We propose a textbfPrompt-to-textbfPrompt generation methodology (textbfP2P) to distill instructive visual prompts for transferable knowledge discovery. The core of P2P is to mine semantic-related instruction from prompt-conditioned visual features and text instruction on modal-sharing semantic concepts.
  arXiv  Detail & Related papers  (2024-06-05T07:59:48Z)
• CLAP4CLIP: Continual Learning with Probabilistic Finetuning for Vision-Language Models [23.398619576886375]
  Continual learning (CL) aims to help deep neural networks learn new knowledge while retaining what has been learned. Our work proposes Continual LeArning with Probabilistic finetuning (CLAP) - a probabilistic modeling framework over visual-guided text features per task.
  arXiv  Detail & Related papers  (2024-03-28T04:15:58Z)
• Few-Shot Class Incremental Learning with Attention-Aware Self-Adaptive Prompt [58.880105981772324]
  We propose a novel framework named Attention-aware Self-adaptive Prompt (ASP) ASP encourages task-invariant prompts to capture shared knowledge by reducing specific information from the attention aspect. In summary, ASP prevents overfitting on base task and does not require enormous data in few-shot incremental tasks.
  arXiv  Detail & Related papers  (2024-03-14T20:34:53Z)
• Semantic Residual Prompts for Continual Learning [21.986800282078498]
  We show that our method significantly outperforms both state-of-the-art CL approaches and the zero-shot CLIP test. Our findings hold true even for datasets with a substantial domain gap w.r.t. the pre-training knowledge of the backbone model.
  arXiv  Detail & Related papers  (2024-03-11T16:23:38Z)
• Learning to Prompt with Text Only Supervision for Vision-Language Models [107.282881515667]
  One branch of methods adapts CLIP by learning prompts using visual information. An alternative approach resorts to training-free methods by generating class descriptions from large language models. We propose to combine the strengths of both streams by learning prompts using only text data.
  arXiv  Detail & Related papers  (2024-01-04T18:59:49Z)
• Self-regulating Prompts: Foundational Model Adaptation without Forgetting [112.66832145320434]
  We introduce a self-regularization framework for prompting called PromptSRC. PromptSRC guides the prompts to optimize for both task-specific and task-agnostic general representations.
  arXiv  Detail & Related papers  (2023-07-13T17:59:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.