Vector-Quantized Input-Contextualized Soft Prompts for Natural Language Understanding

• URL: http://arxiv.org/abs/2205.11024v1
• Date: Mon, 23 May 2022 03:51:27 GMT
• Title: Vector-Quantized Input-Contextualized Soft Prompts for Natural Language Understanding
• Authors: Rishabh Bhardwaj, Amrita Saha, Steven C.H. Hoi
• Abstract summary: We propose a novel way of prompting, Vector-quantized Input-contextualized Prompt Tuning or VIP. Over a wide range of natural language understanding tasks, our proposed VIP framework beats the PT model by a margin of 1.19%.
• Score: 62.45760673220339
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Prompt Tuning (PT) has been largely successful as a parameter-efficient way of conditioning large-scale pre-trained language models towards a downstream task. More recently, soft prompt tuning has aimed to learn a fixed set of task-specific continuous vectors, i.e., soft tokens that remain static across the task samples. However, a fixed prompt may not generalize well to the diverse kinds of inputs the task comprises. With this motivation, we propose a novel way of prompting, Vector-quantized Input-contextualized Prompt Tuning or VIP. Essentially, VIP focuses on two aspects i) input-adaptation: input-specific contextualization of the soft tokens; and ii) vector quantization: we pass the tokens through a quantizer which effectively reduces representation variance by sampling prompts from a compact latent space. Over a wide range of natural language understanding tasks (SuperGLUE, QA, Relation Classification, NER, NLI), our proposed VIP framework beats the PT model by a margin of 1.19\%. Additionally, on Out-of-domain QA and Multi-Task setups over 4 different tasks spanning over 12 domains, we find that VIP outperforms PT by 0.75\%.

Related papers

• VQ-CTAP: Cross-Modal Fine-Grained Sequence Representation Learning for Speech Processing [81.32613443072441]
  For tasks such as text-to-speech (TTS), voice conversion (VC), and automatic speech recognition (ASR), a cross-modal fine-grained (frame-level) sequence representation is desired. We propose a method called Quantized Contrastive Token-Acoustic Pre-training (VQ-CTAP), which uses the cross-modal sequence transcoder to bring text and speech into a joint space.
  arXiv  Detail & Related papers  (2024-08-11T12:24:23Z)
• Task Prompt Vectors: Effective Initialization through Multi-Task Soft-Prompt Transfer [0.6053347262128919]
  We introduce Task Prompt Vectors, created by element-wise difference between weights of tuned soft-prompts and their random initialization. We show that task prompt vectors can be used in low-resource settings to effectively initialize prompt tuning on similar tasks. This allows prompt arithmetics with the pre-trained vectors from different tasks.
  arXiv  Detail & Related papers  (2024-08-02T09:00:03Z)
• MADTP: Multimodal Alignment-Guided Dynamic Token Pruning for Accelerating Vision-Language Transformer [66.71930982549028]
  Vision-Language Transformers (VLTs) have shown great success recently, but are accompanied by heavy computation costs. We propose a novel framework named Multimodal Alignment-Guided Dynamic Token Pruning (MADTP) for accelerating various VLTs.
  arXiv  Detail & Related papers  (2024-03-05T14:13:50Z)
• MPrompt: Exploring Multi-level Prompt Tuning for Machine Reading Comprehension [19.12663587559988]
  We propose a multi-level prompt tuning (MPrompt) method for machine reading comprehension. It utilizes prompts at task-specific, domain-specific, and context-specific levels to enhance the comprehension of input semantics. We conducted extensive experiments on 12 benchmarks of various QA formats and achieved an average improvement of 1.94% over the state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-10-27T14:24:06Z)
• Patch-Prompt Aligned Bayesian Prompt Tuning for Vision-Language Models [48.77653835765705]
  We introduce a probabilistic resolution to prompt tuning, where the label-specific prompts are generated hierarchically by first sampling a latent vector from an underlying distribution and then employing a lightweight generative model. We evaluate the effectiveness of our approach on four tasks: few-shot image recognition, base-to-new generalization, dataset transfer learning, and domain shifts.
  arXiv  Detail & Related papers  (2023-03-16T06:09:15Z)
• Dynamic Prompting: A Unified Framework for Prompt Tuning [33.175097465669374]
  We present a unified dynamic prompt (DP) tuning strategy that dynamically determines different factors of prompts based on specific tasks and instances. Experimental results underscore the significant performance improvement achieved by dynamic prompt tuning across a wide range of tasks. We establish the universal applicability of our approach under full-data, few-shot, and multitask scenarios.
  arXiv  Detail & Related papers  (2023-03-06T06:04:46Z)
• SpeechPrompt v2: Prompt Tuning for Speech Classification Tasks [94.30385972442387]
  We propose SpeechPrompt v2, a prompt tuning framework capable of performing a wide variety of speech classification tasks. Experiment result shows that SpeechPrompt v2 achieves performance on par with prior works with less than 0.15M trainable parameters.
  arXiv  Detail & Related papers  (2023-03-01T18:47:41Z)
• Towards Unified Prompt Tuning for Few-shot Text Classification [47.71344780587704]
  We present the Unified Prompt Tuning (UPT) framework, leading to better few-shot text classification for BERT-style models. In UPT, a novel paradigm Prompt-Options-Verbalizer is proposed for joint prompt learning across different NLP tasks. We also design a self-supervised task named Knowledge-enhanced Selective Masked Language Modeling to improve the PLM's generalization abilities.
  arXiv  Detail & Related papers  (2022-05-11T07:40:45Z)
• Grad2Task: Improved Few-shot Text Classification Using Gradients for Task Representation [24.488427641442694]
  We propose a novel conditional neural process-based approach for few-shot text classification. Our key idea is to represent each task using gradient information from a base model. Our approach outperforms traditional fine-tuning, sequential transfer learning, and state-of-the-art meta learning approaches.
  arXiv  Detail & Related papers  (2022-01-27T15:29:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.