Discriminative Language Model as Semantic Consistency Scorer for Prompt-based Few-Shot Text Classification

• URL: http://arxiv.org/abs/2210.12763v1
• Date: Sun, 23 Oct 2022 16:10:48 GMT
• Title: Discriminative Language Model as Semantic Consistency Scorer for Prompt-based Few-Shot Text Classification
• Authors: Zhipeng Xie and Yahe Li
• Abstract summary: This paper proposes a novel prompt-based finetuning method (called DLM-SCS) for few-shot text classification. The underlying idea is that the prompt instantiated with the true label should have higher semantic consistency score than other prompts with false labels. Our model outperforms several state-of-the-art prompt-based few-shot methods.
• Score: 10.685862129925727
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper proposes a novel prompt-based finetuning method (called DLM-SCS) for few-shot text classification by utilizing the discriminative language model ELECTRA that is pretrained to distinguish whether a token is original or generated. The underlying idea is that the prompt instantiated with the true label should have higher semantic consistency score than other prompts with false labels. Since a prompt usually consists of several components (or parts), its semantic consistency can be decomposed accordingly. The semantic consistency of each component is then computed by making use of the pretrained ELECTRA model, without introducing extra parameters. Extensive experiments have shown that our model outperforms several state-of-the-art prompt-based few-shot methods.

Related papers

• A Fixed-Point Approach to Unified Prompt-Based Counting [51.20608895374113]
  This paper aims to establish a comprehensive prompt-based counting framework capable of generating density maps for objects indicated by various prompt types, such as box, point, and text. Our model excels in prominent class-agnostic datasets and exhibits superior performance in cross-dataset adaptation tasks.
  arXiv  Detail & Related papers  (2024-03-15T12:05:44Z)
• RulePrompt: Weakly Supervised Text Classification with Prompting PLMs and Self-Iterative Logical Rules [30.239044569301534]
  Weakly supervised text classification (WSTC) has attracted increasing attention due to its applicability in classifying a mass of texts. We propose a prompting PLM-based approach named RulePrompt for the WSTC task, consisting of a rule mining module and a rule-enhanced pseudo label generation module. Our approach yields interpretable category rules, proving its advantage in disambiguating easily-confused categories.
  arXiv  Detail & Related papers  (2024-03-05T12:50:36Z)
• MProto: Multi-Prototype Network with Denoised Optimal Transport for Distantly Supervised Named Entity Recognition [75.87566793111066]
  We propose a noise-robust prototype network named MProto for the DS-NER task. MProto represents each entity type with multiple prototypes to characterize the intra-class variance. To mitigate the noise from incomplete labeling, we propose a novel denoised optimal transport (DOT) algorithm.
  arXiv  Detail & Related papers  (2023-10-12T13:02:34Z)
• MetricPrompt: Prompting Model as a Relevance Metric for Few-shot Text Classification [65.51149771074944]
  MetricPrompt eases verbalizer design difficulty by reformulating few-shot text classification task into text pair relevance estimation task. We conduct experiments on three widely used text classification datasets across four few-shot settings. Results show that MetricPrompt outperforms manual verbalizer and other automatic verbalizer design methods across all few-shot settings.
  arXiv  Detail & Related papers  (2023-06-15T06:51:35Z)
• CCPrefix: Counterfactual Contrastive Prefix-Tuning for Many-Class Classification [57.62886091828512]
  We propose a brand-new prefix-tuning method, Counterfactual Contrastive Prefix-tuning (CCPrefix) for many-class classification. Basically, an instance-dependent soft prefix, derived from fact-counterfactual pairs in the label space, is leveraged to complement the language verbalizers in many-class classification.
  arXiv  Detail & Related papers  (2022-11-11T03:45:59Z)
• STPrompt: Semantic-guided and Task-driven prompts for Effective Few-shot Classification [5.6205035780719275]
  We propose the STPrompt -Semantic-guided and Task-driven Prompt model. The proposed model achieves the state-of-the-art performance in five different datasets of few-shot text classification tasks.
  arXiv  Detail & Related papers  (2022-10-29T04:42:30Z)
• Automatic Multi-Label Prompting: Simple and Interpretable Few-Shot Classification [15.575483080819563]
  We propose Automatic Multi-Label Prompting (AMuLaP) to automatically select label mappings for few-shot text classification with prompting. Our method exploits one-to-many label mappings and a statistics-based algorithm to select label mappings given a prompt template. Our experiments demonstrate that AMuLaP achieves competitive performance on the GLUE benchmark without human effort or external resources.
  arXiv  Detail & Related papers  (2022-04-13T11:15:52Z)
• Eliciting Knowledge from Pretrained Language Models for Prototypical Prompt Verbalizer [12.596033546002321]
  In this paper, we focus on eliciting knowledge from pretrained language models and propose a prototypical prompt verbalizer for prompt-tuning. For zero-shot settings, knowledge is elicited from pretrained language models by a manually designed template to form initial prototypical embeddings. For few-shot settings, models are tuned to learn meaningful and interpretable prototypical embeddings.
  arXiv  Detail & Related papers  (2022-01-14T12:04:37Z)
• Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification [68.3291372168167]
  We focus on incorporating external knowledge into the verbalizer, forming a knowledgeable prompt-tuning (KPT) We expand the label word space of the verbalizer using external knowledge bases (KBs) and refine the expanded label word space with the PLM itself before predicting with the expanded label word space. Experiments on zero and few-shot text classification tasks demonstrate the effectiveness of knowledgeable prompt-tuning.
  arXiv  Detail & Related papers  (2021-08-04T13:00:16Z)
• Pre-training Is (Almost) All You Need: An Application to Commonsense Reasoning [61.32992639292889]
  Fine-tuning of pre-trained transformer models has become the standard approach for solving common NLP tasks. We introduce a new scoring method that casts a plausibility ranking task in a full-text format. We show that our method provides a much more stable training phase across random restarts.
  arXiv  Detail & Related papers  (2020-04-29T10:54:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.