Plug-Tagger: A Pluggable Sequence Labeling Framework Using Language Models

• URL: http://arxiv.org/abs/2110.07331v1
• Date: Thu, 14 Oct 2021 13:05:06 GMT
• Title: Plug-Tagger: A Pluggable Sequence Labeling Framework Using Language Models
• Authors: Xin Zhou, Ruotian Ma, Tao Gui, Yiding Tan, Qi Zhang, Xuanjing Huang
• Abstract summary: We propose the use of label word prediction instead of classification for sequence labeling tasks. Our method is up to 70 times faster than non-plug-and-play methods.
• Score: 46.59447116255979
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Plug-and-play functionality allows deep learning models to adapt well to different tasks without requiring any parameters modified. Recently, prefix-tuning was shown to be a plug-and-play method on various text generation tasks by simply inserting corresponding continuous vectors into the inputs. However, sequence labeling tasks invalidate existing plug-and-play methods since different label sets demand changes to the architecture of the model classifier. In this work, we propose the use of label word prediction instead of classification to totally reuse the architecture of pre-trained models for sequence labeling tasks. Specifically, for each task, a label word set is first constructed by selecting a high-frequency word for each class respectively, and then, task-specific vectors are inserted into the inputs and optimized to manipulate the model predictions towards the corresponding label words. As a result, by simply switching the plugin vectors on the input, a frozen pre-trained language model is allowed to perform different tasks. Experimental results on three sequence labeling tasks show that the performance of the proposed method can achieve comparable performance with standard fine-tuning with only 0.1\% task-specific parameters. In addition, our method is up to 70 times faster than non-plug-and-play methods while switching different tasks under the resource-constrained scenario.

Related papers

• ToPro: Token-Level Prompt Decomposition for Cross-Lingual Sequence Labeling Tasks [12.700783525558721]
  ToPro method decomposes an input sentence into single tokens and applies one prompt template to each token. Our experiments on multilingual NER and POS tagging datasets demonstrate that ToPro-based fine-tuning outperforms Vanilla fine-tuning and Prompt-Tuning in zero-shot cross-lingual transfer.
  arXiv  Detail & Related papers  (2024-01-29T21:44:27Z)
• Substituting Data Annotation with Balanced Updates and Collective Loss in Multi-label Text Classification [19.592985329023733]
  Multi-label text classification (MLTC) is the task of assigning multiple labels to a given text. We study the MLTC problem in annotation-free and scarce-annotation settings in which the magnitude of available supervision signals is linear to the number of labels. Our method follows three steps, (1) mapping input text into a set of preliminary label likelihoods by natural language inference using a pre-trained language model, (2) calculating a signed label dependency graph by label descriptions, and (3) updating the preliminary label likelihoods with message passing along the label dependency graph.
  arXiv  Detail & Related papers  (2023-09-24T04:12:52Z)
• MetricPrompt: Prompting Model as a Relevance Metric for Few-shot Text Classification [65.51149771074944]
  MetricPrompt eases verbalizer design difficulty by reformulating few-shot text classification task into text pair relevance estimation task. We conduct experiments on three widely used text classification datasets across four few-shot settings. Results show that MetricPrompt outperforms manual verbalizer and other automatic verbalizer design methods across all few-shot settings.
  arXiv  Detail & Related papers  (2023-06-15T06:51:35Z)
• PIEClass: Weakly-Supervised Text Classification with Prompting and Noise-Robust Iterative Ensemble Training [42.013879670590214]
  Weakly-supervised text classification trains a classifier using the label name of each target class as the only supervision. We propose a new method, PIEClass, consisting of two modules. PIEClass achieves overall better performance than existing strong baselines on seven benchmark datasets.
  arXiv  Detail & Related papers  (2023-05-23T06:19:14Z)
• SepLL: Separating Latent Class Labels from Weak Supervision Noise [4.730767228515796]
  In weakly supervised learning, labeling functions automatically assign, often noisy, labels to data samples. In this work, we provide a method for learning from weak labels by separating two types of complementary information. Our model is competitive with the state-of-the-art, and yields a new best average performance.
  arXiv  Detail & Related papers  (2022-10-25T10:33:45Z)
• Automatic Label Sequence Generation for Prompting Sequence-to-sequence Models [105.4590533269863]
  We propose AutoSeq, a fully automatic prompting method. We adopt natural language prompts on sequence-to-sequence models. Our method reveals the potential of sequence-to-sequence models in few-shot learning.
  arXiv  Detail & Related papers  (2022-09-20T01:35:04Z)
• Improving Multi-task Generalization Ability for Neural Text Matching via Prompt Learning [54.66399120084227]
  Recent state-of-the-art neural text matching models (PLMs) are hard to generalize to different tasks. We adopt a specialization-generalization training strategy and refer to it as Match-Prompt. In specialization stage, descriptions of different matching tasks are mapped to only a few prompt tokens. In generalization stage, text matching model explores the essential matching signals by being trained on diverse multiple matching tasks.
  arXiv  Detail & Related papers  (2022-04-06T11:01:08Z)
• Few-shot Sequence Learning with Transformers [79.87875859408955]
  Few-shot algorithms aim at learning new tasks provided only a handful of training examples. In this work we investigate few-shot learning in the setting where the data points are sequences of tokens. We propose an efficient learning algorithm based on Transformers.
  arXiv  Detail & Related papers  (2020-12-17T12:30:38Z)
• Adaptive Self-training for Few-shot Neural Sequence Labeling [55.43109437200101]
  We develop techniques to address the label scarcity challenge for neural sequence labeling models. Self-training serves as an effective mechanism to learn from large amounts of unlabeled data. meta-learning helps in adaptive sample re-weighting to mitigate error propagation from noisy pseudo-labels.
  arXiv  Detail & Related papers  (2020-10-07T22:29:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.