Introducing Language Guidance in Prompt-based Continual Learning

• URL: http://arxiv.org/abs/2308.15827v1
• Date: Wed, 30 Aug 2023 08:03:49 GMT
• Title: Introducing Language Guidance in Prompt-based Continual Learning
• Authors: Muhammad Gul Zain Ali Khan, Muhammad Ferjad Naeem, Luc Van Gool, Didier Stricker, Federico Tombari, Muhammad Zeshan Afzal
• Abstract summary: We propose Language Guidance for Prompt-based Continual Learning (LGCL) as a plug-in for prompt-based methods. LGCL consistently improves the performance of prompt-based continual learning methods to set a new state-of-the art.
• Score: 95.03110230754423
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Continual Learning aims to learn a single model on a sequence of tasks without having access to data from previous tasks. The biggest challenge in the domain still remains catastrophic forgetting: a loss in performance on seen classes of earlier tasks. Some existing methods rely on an expensive replay buffer to store a chunk of data from previous tasks. This, while promising, becomes expensive when the number of tasks becomes large or data can not be stored for privacy reasons. As an alternative, prompt-based methods have been proposed that store the task information in a learnable prompt pool. This prompt pool instructs a frozen image encoder on how to solve each task. While the model faces a disjoint set of classes in each task in this setting, we argue that these classes can be encoded to the same embedding space of a pre-trained language encoder. In this work, we propose Language Guidance for Prompt-based Continual Learning (LGCL) as a plug-in for prompt-based methods. LGCL is model agnostic and introduces language guidance at the task level in the prompt pool and at the class level on the output feature of the vision encoder. We show with extensive experimentation that LGCL consistently improves the performance of prompt-based continual learning methods to set a new state-of-the art. LGCL achieves these performance improvements without needing any additional learnable parameters.

Related papers

• PECTP: Parameter-Efficient Cross-Task Prompts for Incremental Vision Transformer [76.39111896665585]
  Incremental Learning (IL) aims to learn deep models on sequential tasks continually. Recent vast pre-trained models (PTMs) have achieved outstanding performance by prompt technique in practical IL without the old samples.
  arXiv  Detail & Related papers  (2024-07-04T10:37:58Z)
• Convolutional Prompting meets Language Models for Continual Learning [4.115213208594654]
  Continual Learning (CL) enables machine learning models to learn from continuously shifting new training data in absence of data from old tasks. We propose ConvPrompt, a novel convolutional prompt creation mechanism that maintains layer-wise shared embeddings. The intelligent use of convolution enables us to maintain a low parameter overhead without compromising performance.
  arXiv  Detail & Related papers  (2024-03-29T17:40:37Z)
• Learning to Prompt with Text Only Supervision for Vision-Language Models [107.282881515667]
  One branch of methods adapts CLIP by learning prompts using visual information. An alternative approach resorts to training-free methods by generating class descriptions from large language models. We propose to combine the strengths of both streams by learning prompts using only text data.
  arXiv  Detail & Related papers  (2024-01-04T18:59:49Z)
• When Prompt-based Incremental Learning Does Not Meet Strong Pretraining [36.0889029038102]
  In this work, we develop a learnable Adaptive Prompt Generator (APG) The key is to unify the prompt retrieval and prompt learning processes into a learnable prompt generator. Our method significantly outperforms advanced methods in exemplar-free incremental learning without (strong) pretraining.
  arXiv  Detail & Related papers  (2023-08-21T03:33:21Z)
• Exploring Continual Learning for Code Generation Models [80.78036093054855]
  Continual Learning (CL) is an important aspect that remains underexplored in the code domain. We introduce a benchmark called CodeTask-CL that covers a wide range of tasks, including code generation, translation, summarization, and refinement. We find that effective methods like Prompt Pooling (PP) suffer from catastrophic forgetting due to the unstable training of the prompt selection mechanism.
  arXiv  Detail & Related papers  (2023-07-05T16:58:39Z)
• OverPrompt: Enhancing ChatGPT through Efficient In-Context Learning [49.38867353135258]
  We propose OverPrompt, leveraging the in-context learning capability of LLMs to handle multiple task inputs. Our experiments show that OverPrompt can achieve cost-efficient zero-shot classification without causing significant detriment to task performance.
  arXiv  Detail & Related papers  (2023-05-24T10:08:04Z)
• Instance-wise Prompt Tuning for Pretrained Language Models [72.74916121511662]
  Instance-wise Prompt Tuning (IPT) is the first prompt learning paradigm that injects knowledge from the input data instances to the prompts. IPT significantly outperforms task-based prompt learning methods, and achieves comparable performance to conventional finetuning with only 0.5% - 1.5% of tuned parameters.
  arXiv  Detail & Related papers  (2022-06-04T10:08:50Z)
• Learning to Prompt for Continual Learning [34.609384246149325]
  This work presents a new paradigm for continual learning that aims to train a more succinct memory system without accessing task identity at test time. Our method learns to dynamically prompt (L2P) a pre-trained model to learn tasks sequentially under different task transitions. The objective is to optimize prompts to instruct the model prediction and explicitly manage task-invariant and task-specific knowledge while maintaining model plasticity.
  arXiv  Detail & Related papers  (2021-12-16T06:17:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.