Empowering Sentence Encoders with Prompting and Label Retrieval for Zero-shot Text Classification

• URL: http://arxiv.org/abs/2212.10391v2
• Date: Fri, 19 May 2023 08:37:33 GMT
• Title: Empowering Sentence Encoders with Prompting and Label Retrieval for Zero-shot Text Classification
• Authors: Jimin Hong, Jungsoo Park, Daeyoung Kim, Seongjae Choi, Bokyung Son, and Jaewook Kang
• Abstract summary: Our framework, RaLP, encodes prompted label candidates with a sentence encoder, then assigns the label whose prompt embedding has the highest similarity with the input text embedding. RaLP achieves competitive or stronger performance than much larger baselines on various closed-set classification and multiple-choice QA datasets.
• Score: 5.484132137132862
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: With contrastive pre-training, sentence encoders are generally optimized to locate semantically similar samples closer to each other in their embedding spaces. In this work, we focus on the potential of their embedding spaces to be readily adapted to zero-shot text classification, as semantically distinct samples are already well-separated. Our framework, RaLP (Retrieval augmented Label Prompts for sentence encoder), encodes prompted label candidates with a sentence encoder, then assigns the label whose prompt embedding has the highest similarity with the input text embedding. In order to compensate for the potentially poorly descriptive labels in their original format, RaLP retrieves sentences that are semantically similar to the original label prompt from external corpora and use them as additional pseudo-label prompts. RaLP achieves competitive or stronger performance than much larger baselines on various closed-set classification and multiple-choice QA datasets under zero-shot settings. We show that the retrieval component plays a pivotal role in RaLP's success, and its results are robustly attained regardless of verbalizer variations.

Related papers

• Gen-Z: Generative Zero-Shot Text Classification with Contextualized Label Descriptions [50.92702206798324]
  We propose a generative prompting framework for zero-shot text classification. GEN-Z measures the LM likelihood of input text conditioned on natural language descriptions of labels. We show that zero-shot classification with simple contextualization of the data source consistently outperforms both zero-shot and few-shot baselines.
  arXiv  Detail & Related papers  (2023-11-13T07:12:57Z)
• Rank-Aware Negative Training for Semi-Supervised Text Classification [3.105629960108712]
  Semi-supervised text classification-based paradigms (SSTC) typically employ the spirit of self-training. This paper presents a Rank-aware Negative Training (RNT) framework to address SSTC in learning with noisy label manner.
  arXiv  Detail & Related papers  (2023-06-13T08:41:36Z)
• Exploring Structured Semantic Prior for Multi Label Recognition with Incomplete Labels [60.675714333081466]
  Multi-label recognition (MLR) with incomplete labels is very challenging. Recent works strive to explore the image-to-label correspondence in the vision-language model, ie, CLIP, to compensate for insufficient annotations. We advocate remedying the deficiency of label supervision for the MLR with incomplete labels by deriving a structured semantic prior.
  arXiv  Detail & Related papers  (2023-03-23T12:39:20Z)
• Unsupervised Ranking and Aggregation of Label Descriptions for Zero-Shot Classifiers [8.434227773463022]
  In a true zero-shot setup, designing good label descriptions is challenging because no development set is available. We look at how probabilistic models of repeated rating analysis can be used for selecting the best label descriptions in an unsupervised fashion.
  arXiv  Detail & Related papers  (2022-04-20T14:23:09Z)
• Label Semantic Aware Pre-training for Few-shot Text Classification [53.80908620663974]
  We propose Label Semantic Aware Pre-training (LSAP) to improve the generalization and data efficiency of text classification systems. LSAP incorporates label semantics into pre-trained generative models (T5 in our case) by performing secondary pre-training on labeled sentences from a variety of domains.
  arXiv  Detail & Related papers  (2022-04-14T17:33:34Z)
• Language-driven Semantic Segmentation [88.21498323896475]
  We present LSeg, a novel model for language-driven semantic image segmentation. We use a text encoder to compute embeddings of descriptive input labels. The encoder is trained with a contrastive objective to align pixel embeddings to the text embedding of the corresponding semantic class.
  arXiv  Detail & Related papers  (2022-01-10T18:59:10Z)
• Cross-domain Speech Recognition with Unsupervised Character-level Distribution Matching [60.8427677151492]
  We propose CMatch, a Character-level distribution matching method to perform fine-grained adaptation between each character in two domains. Experiments on the Libri-Adapt dataset show that our proposed approach achieves 14.39% and 16.50% relative Word Error Rate (WER) reduction on both cross-device and cross-environment ASR.
  arXiv  Detail & Related papers  (2021-04-15T14:36:54Z)
• Unsupervised Label Refinement Improves Dataless Text Classification [48.031421660674745]
  Dataless text classification is capable of classifying documents into previously unseen labels by assigning a score to any document paired with a label description. While promising, it crucially relies on accurate descriptions of the label set for each downstream task. This reliance causes dataless classifiers to be highly sensitive to the choice of label descriptions and hinders the broader application of dataless classification in practice.
  arXiv  Detail & Related papers  (2020-12-08T03:37:50Z)
• Label-Wise Document Pre-Training for Multi-Label Text Classification [14.439051753832032]
  This paper develops Label-Wise Pre-Training (LW-PT) method to get a document representation with label-aware information. The basic idea is that, a multi-label document can be represented as a combination of multiple label-wise representations, and that, correlated labels always cooccur in the same or similar documents.
  arXiv  Detail & Related papers  (2020-08-15T10:34:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.