Task Prompt Vectors: Effective Initialization through Multi-Task Soft-Prompt Transfer

• URL: http://arxiv.org/abs/2408.01119v2
• Date: Wed, 23 Oct 2024 14:37:50 GMT
• Title: Task Prompt Vectors: Effective Initialization through Multi-Task Soft-Prompt Transfer
• Authors: Robert Belanec, Simon Ostermann, Ivan Srba, Maria Bielikova,
• Abstract summary: We introduce Task Prompt Vectors, created by element-wise difference between weights of tuned soft-prompts and their random initialization. We show that task prompt vectors can be used in low-resource settings to effectively initialize prompt tuning on similar tasks. This allows prompt arithmetics with the pre-trained vectors from different tasks.
• Score: 0.6053347262128919
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Prompt tuning is an efficient solution for training large language models (LLMs). However, current soft-prompt-based methods often sacrifice multi-task modularity, requiring the training process to be fully or partially repeated for each newly added task. While recent work on task vectors applied arithmetic operations on full model weights to achieve the desired multi-task performance, a similar approach for soft-prompts is still missing. To this end, we introduce Task Prompt Vectors, created by element-wise difference between weights of tuned soft-prompts and their random initialization. Experimental results on 12 NLU datasets show that task prompt vectors can be used in low-resource settings to effectively initialize prompt tuning on similar tasks. In addition, we show that task prompt vectors are independent of the random initialization of prompt tuning on 2 different language model architectures. This allows prompt arithmetics with the pre-trained vectors from different tasks. In this way, we provide a competitive alternative to state-of-the-art baselines by arithmetic addition of task prompt vectors from multiple tasks.

Related papers

• Task Facet Learning: A Structured Approach to Prompt Optimization [14.223730629357178]
  We propose an algorithm that learns multiple facets of a task from a set of training examples. The resulting algorithm, UniPrompt, consists of a generative model to generate initial candidates for each prompt section. Empirical evaluation on multiple datasets and a real-world task shows that prompts generated using UniPrompt obtain higher accuracy than human-tuned prompts.
  arXiv  Detail & Related papers  (2024-06-15T04:54:26Z)
• Less is more: Summarizing Patch Tokens for efficient Multi-Label Class-Incremental Learning [38.36863497458095]
  We propose a new class-incremental learning method Multi-Label class incremental learning via summarising pAtch tokeN Embeddings (MULTI-LANE) Our proposed method Multi-Label class incremental learning via summarising pAtch tokeN Embeddings (MULTI-LANE) enables learning disentangled task-specific representations in MLCIL while ensuring fast inference.
  arXiv  Detail & Related papers  (2024-05-24T15:18:27Z)
• TransPrompt v2: A Transferable Prompting Framework for Cross-task Text Classification [37.824031151922604]
  We propose TransPrompt v2, a novel transferable prompting framework for few-shot learning across similar or distant text classification tasks. For learning across similar tasks, we employ a multi-task meta-knowledge acquisition (MMA) procedure to train a meta-learner. For learning across distant tasks, we inject the task type descriptions into the prompt, and capture the intra-type and inter-type prompt embeddings.
  arXiv  Detail & Related papers  (2023-08-29T04:16:57Z)
• Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learning [43.639430661322585]
  We propose multitask prompt tuning (MPT) MPT learns a single transferable prompt by distilling knowledge from multiple task-specific source prompts. We then learn multiplicative low rank updates to this shared prompt to efficiently adapt it to each downstream target task.
  arXiv  Detail & Related papers  (2023-03-06T03:25:59Z)
• Multitask Vision-Language Prompt Tuning [103.5967011236282]
  We propose multitask vision-language prompt tuning (MV) MV incorporates cross-task knowledge into prompt tuning for vision-language models. Results in 20 vision tasks demonstrate that the proposed approach outperforms all single-task baseline prompt tuning methods.
  arXiv  Detail & Related papers  (2022-11-21T18:41:44Z)
• Multitask Pre-training of Modular Prompt for Chinese Few-Shot Learning [83.10861551885321]
  We present Multi-task Pre-trained Modular Prompt (MP2) to boost prompt tuning for few-shot learning. MP2 is a set of combinable prompts pre-trained on 38 Chinese tasks. We show MP2 significantly outperforms prompt tuning, full model tuning, and prior prompt pre-training methods in few-shot settings.
  arXiv  Detail & Related papers  (2022-10-14T06:43:42Z)
• Decomposed Prompting: A Modular Approach for Solving Complex Tasks [55.42850359286304]
  We propose Decomposed Prompting to solve complex tasks by decomposing them (via prompting) into simpler sub-tasks. This modular structure allows each prompt to be optimized for its specific sub-task. We show that the flexibility and modularity of Decomposed Prompting allows it to outperform prior work on few-shot prompting.
  arXiv  Detail & Related papers  (2022-10-05T17:28:20Z)
• Instance-wise Prompt Tuning for Pretrained Language Models [72.74916121511662]
  Instance-wise Prompt Tuning (IPT) is the first prompt learning paradigm that injects knowledge from the input data instances to the prompts. IPT significantly outperforms task-based prompt learning methods, and achieves comparable performance to conventional finetuning with only 0.5% - 1.5% of tuned parameters.
  arXiv  Detail & Related papers  (2022-06-04T10:08:50Z)
• Attentional Mixtures of Soft Prompt Tuning for Parameter-efficient Multi-task Knowledge Sharing [53.399742232323895]
  ATTEMPT is a new modular, multi-task, and parameter-efficient language model (LM) tuning approach. It combines knowledge transferred across different tasks via a mixture of soft prompts while keeping original LM unchanged. It is parameter-efficient (e.g., updates 1,600 times fewer parameters than fine-tuning) and enables multi-task learning and flexible extensions.
  arXiv  Detail & Related papers  (2022-05-24T10:48:33Z)
• Vector-Quantized Input-Contextualized Soft Prompts for Natural Language Understanding [62.45760673220339]
  We propose a novel way of prompting, Vector-quantized Input-contextualized Prompt Tuning or VIP. Over a wide range of natural language understanding tasks, our proposed VIP framework beats the PT model by a margin of 1.19%.
  arXiv  Detail & Related papers  (2022-05-23T03:51:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.