Prototypical Verbalizer for Prompt-based Few-shot Tuning

• URL: http://arxiv.org/abs/2203.09770v1
• Date: Fri, 18 Mar 2022 07:07:56 GMT
• Title: Prototypical Verbalizer for Prompt-based Few-shot Tuning
• Authors: Ganqu Cui, Shengding Hu, Ning Ding, Longtao Huang, Zhiyuan Liu
• Abstract summary: We propose the verbalizer (ProtoVerb) which is built directly from training data. ProtoVerb learns prototype vectors as prototypical verbalizers by contrastive learning. We conduct experiments on both topic classification and entity typing tasks, and the results demonstrate that ProtoVerb significantly outperforms current automatic verbalizers.
• Score: 32.74024339482436
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Prompt-based tuning for pre-trained language models (PLMs) has shown its effectiveness in few-shot learning. Typically, prompt-based tuning wraps the input text into a cloze question. To make predictions, the model maps the output words to labels via a verbalizer, which is either manually designed or automatically built. However, manual verbalizers heavily depend on domain-specific prior knowledge and human efforts, while finding appropriate label words automatically still remains challenging.In this work, we propose the prototypical verbalizer (ProtoVerb) which is built directly from training data. Specifically, ProtoVerb learns prototype vectors as verbalizers by contrastive learning. In this way, the prototypes summarize training instances and are able to enclose rich class-level semantics. We conduct experiments on both topic classification and entity typing tasks, and the results demonstrate that ProtoVerb significantly outperforms current automatic verbalizers, especially when training data is extremely scarce. More surprisingly, ProtoVerb consistently boosts prompt-based tuning even on untuned PLMs, indicating an elegant non-tuning way to utilize PLMs. Our codes are avaliable at https://github.com/thunlp/OpenPrompt.

Related papers

• Evolutionary Verbalizer Search for Prompt-based Few Shot Text Classification [5.583948835737293]
  We propose a novel evolutionary verbalizer search (EVS) algorithm to improve prompt-based tuning with the high-performance verbalizer. In this paper, we focus on automatically constructing the optimal verbalizer and propose a novelEVS algorithm to improve prompt-based tuning with the high-performance verbalizer.
  arXiv  Detail & Related papers  (2023-06-18T10:03:11Z)
• Assessing Phrase Break of ESL Speech with Pre-trained Language Models and Large Language Models [7.782346535009883]
  This work introduces approaches to assessing phrase breaks in ESL learners' speech using pre-trained language models (PLMs) and large language models (LLMs)
  arXiv  Detail & Related papers  (2023-06-08T07:10:39Z)
• Prompt Tuning for Discriminative Pre-trained Language Models [96.04765512463415]
  Recent works have shown promising results of prompt tuning in stimulating pre-trained language models (PLMs) for natural language processing (NLP) tasks. It is still unknown whether and how discriminative PLMs, e.g., ELECTRA, can be effectively prompt-tuned. We present DPT, the first prompt tuning framework for discriminative PLMs, which reformulates NLP tasks into a discriminative language modeling problem.
  arXiv  Detail & Related papers  (2022-05-23T10:11:50Z)
• PromptDA: Label-guided Data Augmentation for Prompt-based Few-shot Learners [15.130992223266734]
  We propose a novel label-guided data augmentation framework, PromptDA, which exploits the enriched label semantic information for data augmentation. Our experiment results on few-shot text classification tasks demonstrate the superior performance of the proposed framework.
  arXiv  Detail & Related papers  (2022-05-18T22:15:20Z)
• Wav2Seq: Pre-training Speech-to-Text Encoder-Decoder Models Using Pseudo Languages [58.43299730989809]
  We introduce Wav2Seq, the first self-supervised approach to pre-train both parts of encoder-decoder models for speech data. We induce a pseudo language as a compact discrete representation, and formulate a self-supervised pseudo speech recognition task. This process stands on its own, or can be applied as low-cost second-stage pre-training.
  arXiv  Detail & Related papers  (2022-05-02T17:59:02Z)
• ProtoTEx: Explaining Model Decisions with Prototype Tensors [27.779971257213553]
  ProtoTEx is a novel white-box NLP classification architecture based on prototype networks. We describe a novel interleaved training algorithm that effectively handles classes characterized by the absence of indicative features. On a propaganda detection task, ProtoTEx accuracy matches BART-large and exceeds BERT-large with the added benefit of providing faithful explanations.
  arXiv  Detail & Related papers  (2022-04-11T22:08:45Z)
• PERFECT: Prompt-free and Efficient Few-shot Learning with Language Models [67.3725459417758]
  PERFECT is a simple and efficient method for few-shot fine-tuning of PLMs without relying on any such handcrafting. We show that manually engineered task prompts can be replaced with task-specific adapters that enable sample-efficient fine-tuning. Experiments on a wide range of few-shot NLP tasks demonstrate that PERFECT, while being simple and efficient, also outperforms existing state-of-the-art few-shot learning methods.
  arXiv  Detail & Related papers  (2022-04-03T22:31:25Z)
• Eliciting Knowledge from Pretrained Language Models for Prototypical Prompt Verbalizer [12.596033546002321]
  In this paper, we focus on eliciting knowledge from pretrained language models and propose a prototypical prompt verbalizer for prompt-tuning. For zero-shot settings, knowledge is elicited from pretrained language models by a manually designed template to form initial prototypical embeddings. For few-shot settings, models are tuned to learn meaningful and interpretable prototypical embeddings.
  arXiv  Detail & Related papers  (2022-01-14T12:04:37Z)
• Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification [68.3291372168167]
  We focus on incorporating external knowledge into the verbalizer, forming a knowledgeable prompt-tuning (KPT) We expand the label word space of the verbalizer using external knowledge bases (KBs) and refine the expanded label word space with the PLM itself before predicting with the expanded label word space. Experiments on zero and few-shot text classification tasks demonstrate the effectiveness of knowledgeable prompt-tuning.
  arXiv  Detail & Related papers  (2021-08-04T13:00:16Z)
• PTR: Prompt Tuning with Rules for Text Classification [64.1655047016891]
  Fine-tuned pre-trained language models (PLMs) have achieved awesome performance on almost all NLP tasks. We propose prompt tuning with rules (PTR) for many-class text classification. PTR is able to encode prior knowledge of each class into prompt tuning.
  arXiv  Detail & Related papers  (2021-05-24T13:24:02Z)
• Fast End-to-End Speech Recognition via a Non-Autoregressive Model and Cross-Modal Knowledge Transferring from BERT [72.93855288283059]
  We propose a non-autoregressive speech recognition model called LASO (Listen Attentively, and Spell Once) The model consists of an encoder, a decoder, and a position dependent summarizer (PDS)
  arXiv  Detail & Related papers  (2021-02-15T15:18:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.