Automatic Label Sequence Generation for Prompting Sequence-to-sequence Models

• URL: http://arxiv.org/abs/2209.09401v1
• Date: Tue, 20 Sep 2022 01:35:04 GMT
• Title: Automatic Label Sequence Generation for Prompting Sequence-to-sequence Models
• Authors: Zichun Yu, Tianyu Gao, Zhengyan Zhang, Yankai Lin, Zhiyuan Liu, Maosong Sun and Jie Zhou
• Abstract summary: We propose AutoSeq, a fully automatic prompting method. We adopt natural language prompts on sequence-to-sequence models. Our method reveals the potential of sequence-to-sequence models in few-shot learning.
• Score: 105.4590533269863
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Prompting, which casts downstream applications as language modeling tasks, has shown to be sample efficient compared to standard fine-tuning with pre-trained models. However, one pitfall of prompting is the need of manually-designed patterns, whose outcome can be unintuitive and requires large validation sets to tune. To tackle the challenge, we propose AutoSeq, a fully automatic prompting method: (1) We adopt natural language prompts on sequence-to-sequence models, enabling free-form generation and larger label search space; (2) We propose label sequences -- phrases with indefinite lengths to verbalize the labels -- which eliminate the need of manual templates and are more expressive than single label words; (3) We use beam search to automatically generate a large amount of label sequence candidates and propose contrastive re-ranking to get the best combinations. AutoSeq significantly outperforms other no-manual-design methods, such as soft prompt tuning, adapter tuning, and automatic search on single label words; the generated label sequences are even better than curated manual ones on a variety of tasks. Our method reveals the potential of sequence-to-sequence models in few-shot learning and sheds light on a path to generic and automatic prompting. The source code of this paper can be obtained from https://github.com/thunlp/Seq2Seq-Prompt.

Related papers

• Automated Black-box Prompt Engineering for Personalized Text-to-Image Generation [150.57983348059528]
  PRISM is an algorithm that automatically identifies human-interpretable and transferable prompts. It can effectively generate desired concepts given only black-box access to T2I models. Our experiments demonstrate the versatility and effectiveness of PRISM in generating accurate prompts for objects, styles and images.
  arXiv  Detail & Related papers  (2024-03-28T02:35:53Z)
• OTSeq2Set: An Optimal Transport Enhanced Sequence-to-Set Model for Extreme Multi-label Text Classification [9.990725102725916]
  Extreme multi-label text classification (XMTC) is the task of finding the most relevant subset labels from a large-scale label collection. We propose an autoregressive sequence-to-set model for XMTC tasks named OTSeq2Set. Our model generates predictions in student-forcing scheme and is trained by a loss function based on bipartite matching.
  arXiv  Detail & Related papers  (2022-10-26T07:25:18Z)
• Modeling sequential annotations for sequence labeling with crowds [8.239028141030621]
  Crowd sequential annotations can be an efficient and cost-effective way to build large datasets for sequence labeling. We propose Modeling sequential annotation for sequence labeling with crowds (SA-SLC) A valid label sequence inference (VLSE) method is proposed to derive the valid ground-truth label sequences from crowd sequential annotations.
  arXiv  Detail & Related papers  (2022-09-20T02:51:23Z)
• Conditional set generation using Seq2seq models [52.516563721766445]
  Conditional set generation learns a mapping from an input sequence of tokens to a set. Sequence-to-sequence(Seq2seq) models are a popular choice to model set generation. We propose a novel algorithm for effectively sampling informative orders over the space of label orders.
  arXiv  Detail & Related papers  (2022-05-25T04:17:50Z)
• Automatic Multi-Label Prompting: Simple and Interpretable Few-Shot Classification [15.575483080819563]
  We propose Automatic Multi-Label Prompting (AMuLaP) to automatically select label mappings for few-shot text classification with prompting. Our method exploits one-to-many label mappings and a statistics-based algorithm to select label mappings given a prompt template. Our experiments demonstrate that AMuLaP achieves competitive performance on the GLUE benchmark without human effort or external resources.
  arXiv  Detail & Related papers  (2022-04-13T11:15:52Z)
• Plug-Tagger: A Pluggable Sequence Labeling Framework Using Language Models [46.59447116255979]
  We propose the use of label word prediction instead of classification for sequence labeling tasks. Our method is up to 70 times faster than non-plug-and-play methods.
  arXiv  Detail & Related papers  (2021-10-14T13:05:06Z)
• Accelerating BERT Inference for Sequence Labeling via Early-Exit [65.7292767360083]
  We extend the recent successful early-exit mechanism to accelerate the inference of PTMs for sequence labeling tasks. We also propose a token-level early-exit mechanism that allows partial tokens to exit early at different layers. Our approach can save up to 66%-75% inference cost with minimal performance degradation.
  arXiv  Detail & Related papers  (2021-05-28T14:39:26Z)
• SeqMix: Augmenting Active Sequence Labeling via Sequence Mixup [11.606681893887604]
  We propose a simple but effective data augmentation method to improve the label efficiency of active sequence labeling. Our method, SeqMix, simply augments the queried samples by generating extra labeled sequences in each iteration. In SeqMix, we address this challenge by performing mixup for both sequences and token-level labels of the queried samples.
  arXiv  Detail & Related papers  (2020-10-05T20:27:14Z)
• Interaction Matching for Long-Tail Multi-Label Classification [57.262792333593644]
  We present an elegant and effective approach for addressing limitations in existing multi-label classification models. By performing soft n-gram interaction matching, we match labels with natural language descriptions.
  arXiv  Detail & Related papers  (2020-05-18T15:27:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.