Semantic Prompting with Image-Token for Continual Learning

• URL: http://arxiv.org/abs/2403.11537v1
• Date: Mon, 18 Mar 2024 07:43:14 GMT
• Title: Semantic Prompting with Image-Token for Continual Learning
• Authors: Jisu Han, Jaemin Na, Wonjun Hwang,
• Abstract summary: I-Prompt is a task-agnostic approach to eliminate task prediction. Our method achieves competitive performance on four benchmarks. We demonstrate the superiority of our method across various scenarios through extensive experiments.
• Score: 7.5140668729696145
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Continual learning aims to refine model parameters for new tasks while retaining knowledge from previous tasks. Recently, prompt-based learning has emerged to leverage pre-trained models to be prompted to learn subsequent tasks without the reliance on the rehearsal buffer. Although this approach has demonstrated outstanding results, existing methods depend on preceding task-selection process to choose appropriate prompts. However, imperfectness in task-selection may lead to negative impacts on the performance particularly in the scenarios where the number of tasks is large or task distributions are imbalanced. To address this issue, we introduce I-Prompt, a task-agnostic approach focuses on the visual semantic information of image tokens to eliminate task prediction. Our method consists of semantic prompt matching, which determines prompts based on similarities between tokens, and image token-level prompting, which applies prompts directly to image tokens in the intermediate layers. Consequently, our method achieves competitive performance on four benchmarks while significantly reducing training time compared to state-of-the-art methods. Moreover, we demonstrate the superiority of our method across various scenarios through extensive experiments.

Related papers

• CAPrompt: Cyclic Prompt Aggregation for Pre-Trained Model Based Class Incremental Learning [12.249938312431993]
  We propose a novel Cyclic Prompt Aggregation (CAPrompt) method to eliminate the dependency on task ID prediction. Under concave conditions, the aggregated prompt achieves lower error compared to selecting a single task-specific prompt. Our proposed CAPrompt outperforms state-of-the-art methods by 2%-3%.
  arXiv  Detail & Related papers  (2024-12-12T04:34:28Z)
• Adaptive Prompting for Continual Relation Extraction: A Within-Task Variance Perspective [23.79259400522239]
  We propose a novel approach to address catastrophic forgetting in Continual Relation Extraction. Our approach employs a prompt pool for each task, capturing variations within each task while enhancing cross-task variances.
  arXiv  Detail & Related papers  (2024-12-11T11:00:33Z)
• Vector Quantization Prompting for Continual Learning [23.26682439914273]
  Continual learning requires to overcome catastrophic forgetting when training a single model on a sequence of tasks. Recent top-performing approaches are prompt-based methods that utilize a set of learnable parameters to encode task knowledge. We propose VQ-Prompt, a prompt-based continual learning method that incorporates Vector Quantization into end-to-end training of a set of discrete prompts.
  arXiv  Detail & Related papers  (2024-10-27T13:43:53Z)
• Data-CUBE: Data Curriculum for Instruction-based Sentence Representation Learning [85.66907881270785]
  We propose a data curriculum method, namely Data-CUBE, that arranges the orders of all the multi-task data for training. In the task level, we aim to find the optimal task order to minimize the total cross-task interference risk. In the instance level, we measure the difficulty of all instances per task, then divide them into the easy-to-difficult mini-batches for training.
  arXiv  Detail & Related papers  (2024-01-07T18:12:20Z)
• Active Instruction Tuning: Improving Cross-Task Generalization by Training on Prompt Sensitive Tasks [101.40633115037983]
  Instruction tuning (IT) achieves impressive zero-shot generalization results by training large language models (LLMs) on a massive amount of diverse tasks with instructions. How to select new tasks to improve the performance and generalizability of IT models remains an open question. We propose active instruction tuning based on prompt uncertainty, a novel framework to identify informative tasks, and then actively tune the models on the selected tasks.
  arXiv  Detail & Related papers  (2023-11-01T04:40:05Z)
• Continual Prompt Tuning for Dialog State Tracking [58.66412648276873]
  A desirable dialog system should be able to continually learn new skills without forgetting old ones. We present Continual Prompt Tuning, a parameter-efficient framework that not only avoids forgetting but also enables knowledge transfer between tasks.
  arXiv  Detail & Related papers  (2022-03-13T13:22:41Z)
• PROMPT WAYWARDNESS: The Curious Case of Discretized Interpretation of Continuous Prompts [99.03864962014431]
  Fine-tuning continuous prompts for target tasks has emerged as a compact alternative to full model fine-tuning. In practice, we observe a "wayward" behavior between the task solved by continuous prompts and their nearest neighbor.
  arXiv  Detail & Related papers  (2021-12-15T18:55:05Z)
• Pretext Tasks selection for multitask self-supervised speech representation learning [23.39079406674442]
  This paper introduces a method to select a group of pretext tasks among a set of candidates. Experiments conducted on speaker recognition and automatic speech recognition validate our approach.
  arXiv  Detail & Related papers  (2021-07-01T16:36:29Z)
• Conditional Meta-Learning of Linear Representations [57.90025697492041]
  Standard meta-learning for representation learning aims to find a common representation to be shared across multiple tasks. In this work we overcome this issue by inferring a conditioning function, mapping the tasks' side information into a representation tailored to the task at hand. We propose a meta-algorithm capable of leveraging this advantage in practice.
  arXiv  Detail & Related papers  (2021-03-30T12:02:14Z)
• Learning Invariant Representation for Continual Learning [5.979373021392084]
  A key challenge in Continual learning is catastrophically forgetting previously learned tasks when the agent faces a new one. We propose a new pseudo-rehearsal-based method, named learning Invariant Representation for Continual Learning (IRCL) Disentangling the shared invariant representation helps to learn continually a sequence of tasks, while being more robust to forgetting and having better knowledge transfer.
  arXiv  Detail & Related papers  (2021-01-15T15:12:51Z)
• Adaptive Task Sampling for Meta-Learning [79.61146834134459]
  Key idea of meta-learning for few-shot classification is to mimic the few-shot situations faced at test time. We propose an adaptive task sampling method to improve the generalization performance.
  arXiv  Detail & Related papers  (2020-07-17T03:15:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.