SPT: Semi-Parametric Prompt Tuning for Multitask Prompted Learning

• URL: http://arxiv.org/abs/2212.10929v1
• Date: Wed, 21 Dec 2022 11:18:09 GMT
• Title: SPT: Semi-Parametric Prompt Tuning for Multitask Prompted Learning
• Authors: M Saiful Bari, Aston Zhang, Shuai Zheng, Xingjian Shi, Yi Zhu, Shafiq Joty, Mu Li
• Abstract summary: Multitask prompted learning can help generalization through a diverse set of tasks at once. We propose SPT, a semi-parametric prompt tuning method for multitask prompted learning.
• Score: 28.29889045842277
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Pre-trained large language models can efficiently interpolate human-written prompts in a natural way. Multitask prompted learning can help generalization through a diverse set of tasks at once, thus enhancing the potential for more effective downstream fine-tuning. To perform efficient multitask-inference in the same batch, parameter-efficient fine-tuning methods such as prompt tuning have been proposed. However, the existing prompt tuning methods may lack generalization. We propose SPT, a semi-parametric prompt tuning method for multitask prompted learning. The novel component of SPT is a memory bank from where memory prompts are retrieved based on discrete prompts. Extensive experiments, such as (i) fine-tuning a full language model with SPT on 31 different tasks from 8 different domains and evaluating zero-shot generalization on 9 heldout datasets under 5 NLP task categories and (ii) pretraining SPT on the GLUE datasets and evaluating fine-tuning on the SuperGLUE datasets, demonstrate effectiveness of SPT.

Related papers

• Approximated Prompt Tuning for Vision-Language Pre-trained Models [54.326232586461614]
  In vision-language pre-trained models, prompt tuning often requires a large number of learnable tokens to bridge the gap between the pre-training and downstream tasks. We propose a novel Approximated Prompt Tuning (APT) approach towards efficient VL transfer learning.
  arXiv  Detail & Related papers  (2023-06-27T05:43:47Z)
• Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learning [43.639430661322585]
  We propose multitask prompt tuning (MPT) MPT learns a single transferable prompt by distilling knowledge from multiple task-specific source prompts. We then learn multiplicative low rank updates to this shared prompt to efficiently adapt it to each downstream target task.
  arXiv  Detail & Related papers  (2023-03-06T03:25:59Z)
• How Does In-Context Learning Help Prompt Tuning? [55.78535874154915]
  Fine-tuning large language models is becoming ever more impractical due to their rapidly-growing scale. This motivates the use of parameter-efficient adaptation methods such as prompt tuning (PT), which adds a small number of tunable embeddings to an otherwise frozen model. Recently, Singhal et al. (2022) propose instruction prompt tuning'' (IPT), which combines PT with ICL by concatenating a natural language demonstration with learned prompt embeddings.
  arXiv  Detail & Related papers  (2023-02-22T17:45:12Z)
• Instance-wise Prompt Tuning for Pretrained Language Models [72.74916121511662]
  Instance-wise Prompt Tuning (IPT) is the first prompt learning paradigm that injects knowledge from the input data instances to the prompts. IPT significantly outperforms task-based prompt learning methods, and achieves comparable performance to conventional finetuning with only 0.5% - 1.5% of tuned parameters.
  arXiv  Detail & Related papers  (2022-06-04T10:08:50Z)
• Attentional Mixtures of Soft Prompt Tuning for Parameter-efficient Multi-task Knowledge Sharing [53.399742232323895]
  ATTEMPT is a new modular, multi-task, and parameter-efficient language model (LM) tuning approach. It combines knowledge transferred across different tasks via a mixture of soft prompts while keeping original LM unchanged. It is parameter-efficient (e.g., updates 1,600 times fewer parameters than fine-tuning) and enables multi-task learning and flexible extensions.
  arXiv  Detail & Related papers  (2022-05-24T10:48:33Z)
• Towards Unified Prompt Tuning for Few-shot Text Classification [47.71344780587704]
  We present the Unified Prompt Tuning (UPT) framework, leading to better few-shot text classification for BERT-style models. In UPT, a novel paradigm Prompt-Options-Verbalizer is proposed for joint prompt learning across different NLP tasks. We also design a self-supervised task named Knowledge-enhanced Selective Masked Language Modeling to improve the PLM's generalization abilities.
  arXiv  Detail & Related papers  (2022-05-11T07:40:45Z)
• IDPG: An Instance-Dependent Prompt Generation Method [58.45110542003139]
  Prompt tuning is a new, efficient NLP transfer learning paradigm that adds a task-specific prompt in each input instance during the model training stage. We propose a conditional prompt generation method to generate prompts for each input instance.
  arXiv  Detail & Related papers  (2022-04-09T15:45:27Z)
• Exploring Low-dimensional Intrinsic Task Subspace via Prompt Tuning [70.76016793057283]
  In this work, we study how pre-trained language models (PLMs) learn universal representations and effectively adapt to broad NLP tasks differing a lot. In experiments, we study diverse few-shot NLP tasks and surprisingly find that in a 5-dimensional subspace found with 100 random tasks, by only tuning 5 free parameters, we can recover 87% and 65% of the full prompt tuning performance.
  arXiv  Detail & Related papers  (2021-10-15T05:43:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.