Global Prompt Cell: A Portable Control Module for Effective Prompt Tuning

• URL: http://arxiv.org/abs/2304.05642v2
• Date: Sat, 13 May 2023 07:45:59 GMT
• Title: Global Prompt Cell: A Portable Control Module for Effective Prompt Tuning
• Authors: Chi Liu, Haochun Wang, Nuwa Xi, Sendong Zhao, Bing Qin
• Abstract summary: We introduce the Global Prompt Cell (GPC), a portable control module for prompt tuning. Our experimental results demonstrate a 5.8% improvement on SuperGLUE datasets compared to vanilla prompt tuning.
• Score: 16.76984489127912
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As a novel approach to tuning pre-trained models, prompt tuning involves freezing the parameters in downstream tasks while inserting trainable embeddings into inputs in the first layer. However, previous methods have mainly focused on the initialization of prompt embeddings. The strategy of training and utilizing prompt embeddings in a reasonable way has become a limiting factor in the effectiveness of prompt tuning. To address this issue, we introduce the Global Prompt Cell (GPC), a portable control module for prompt tuning that selectively preserves prompt information across all encoder layers. Our experimental results demonstrate a 5.8% improvement on SuperGLUE datasets compared to vanilla prompt tuning.

Related papers

• Attention Prompt Tuning: Parameter-efficient Adaptation of Pre-trained Models for Spatiotemporal Modeling [32.603558214472265]
  We introduce Attention Prompt Tuning (APT) for video-based applications such as action recognition. APT involves injecting a set of learnable prompts along with data tokens during fine-tuning while keeping the backbone frozen. The proposed approach greatly reduces the number of FLOPs and latency while achieving a significant performance boost.
  arXiv  Detail & Related papers  (2024-03-11T17:59:41Z)
• Improving Prompt Tuning with Learned Prompting Layers [12.46460062708119]
  We propose a novel framework, underlineSelective underlinePrompt underlineTuning (SPT) It learns to select the proper prompt layers by inserting a prompt controlled by a learnable probabilistic gate at each intermediate layer. We conduct extensive experiments with ten benchmark datasets under the full-data and few-shot scenarios.
  arXiv  Detail & Related papers  (2023-10-31T02:07:51Z)
• Gradient-Regulated Meta-Prompt Learning for Generalizable Vision-Language Models [137.74524357614285]
  We introduce a novel Gradient-RegulAted Meta-prompt learning framework. It helps pre-training models adapt to downstream tasks in a parameter -- and data -- efficient way. GRAM can be easily incorporated into various prompt tuning methods in a model-agnostic way.
  arXiv  Detail & Related papers  (2023-03-12T05:03:37Z)
• Late Prompt Tuning: A Late Prompt Could Be Better Than Many Prompts [97.20933523766182]
  Prompt tuning is a parameter-efficient tuning (PETuning) method for utilizing pre-trained models (PTMs) We present Late Prompt Tuning () that inserts a late prompt into an intermediate layer of the PTM instead of the input layer or all layers. We show that, can achieve competitive performance to full model tuning and other PETuning methods under both full-data and few-shot scenarios.
  arXiv  Detail & Related papers  (2022-10-20T14:23:52Z)
• Multitask Pre-training of Modular Prompt for Chinese Few-Shot Learning [83.10861551885321]
  We present Multi-task Pre-trained Modular Prompt (MP2) to boost prompt tuning for few-shot learning. MP2 is a set of combinable prompts pre-trained on 38 Chinese tasks. We show MP2 significantly outperforms prompt tuning, full model tuning, and prior prompt pre-training methods in few-shot settings.
  arXiv  Detail & Related papers  (2022-10-14T06:43:42Z)
• Instance-wise Prompt Tuning for Pretrained Language Models [72.74916121511662]
  Instance-wise Prompt Tuning (IPT) is the first prompt learning paradigm that injects knowledge from the input data instances to the prompts. IPT significantly outperforms task-based prompt learning methods, and achieves comparable performance to conventional finetuning with only 0.5% - 1.5% of tuned parameters.
  arXiv  Detail & Related papers  (2022-06-04T10:08:50Z)
• IDPG: An Instance-Dependent Prompt Generation Method [58.45110542003139]
  Prompt tuning is a new, efficient NLP transfer learning paradigm that adds a task-specific prompt in each input instance during the model training stage. We propose a conditional prompt generation method to generate prompts for each input instance.
  arXiv  Detail & Related papers  (2022-04-09T15:45:27Z)
• PPT: Pre-trained Prompt Tuning for Few-shot Learning [47.05554619258627]
  Prompts for pre-trained language models (PLMs) have shown remarkable performance by bridging the gap between pre-training tasks and various downstream tasks. Among these methods, prompt tuning, which freezes PLMs and only tunes soft prompts, provides an efficient and effective solution for adapting large-scale PLMs to downstream tasks. In our work, we find that prompt tuning performs comparably with conventional full-model fine-tuning when downstream data are sufficient, whereas it performs much worse under few-shot learning settings.
  arXiv  Detail & Related papers  (2021-09-09T15:11:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.