How Many Data Points is a Prompt Worth?

• URL: http://arxiv.org/abs/2103.08493v1
• Date: Tue, 6 Apr 2021 14:25:19 GMT
• Title: How Many Data Points is a Prompt Worth?
• Authors: Teven Le Scao and Alexander M. Rush
• Abstract summary: Proponents of prompting argue that it provides a method for injecting task-specific guidance. We compare prompted and head-based fine-tuning in equal conditions across many tasks and data sizes. Results show that prompting is often worth 100s of data points on average across classification tasks.
• Score: 106.76346863035786
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: When fine-tuning pretrained models for classification, researchers either use a generic model head or a task-specific prompt for prediction. Proponents of prompting have argued that prompts provide a method for injecting task-specific guidance, which is beneficial in low-data regimes. We aim to quantify this benefit through rigorous testing of prompts in a fair setting: comparing prompted and head-based fine-tuning in equal conditions across many tasks and data sizes. By controlling for many sources of advantage, we find that prompting does indeed provide a benefit, and that this benefit can be quantified per task. Results show that prompting is often worth 100s of data points on average across classification tasks.

Related papers

• Task Adaptive Feature Distribution Based Network for Few-shot Fine-grained Target Classification [16.575362884459963]
  We propose TAFD-Net: a task adaptive feature distribution network. It features a task-adaptive component for embedding to capture task-level nuances, an asymmetric metric for calculating feature distribution similarities between query samples and support categories, and a contrastive measure strategy to boost performance.
  arXiv  Detail & Related papers  (2024-10-13T10:56:09Z)
• Bayesian Multi-Task Transfer Learning for Soft Prompt Tuning [44.43258626098661]
  We argue that when we extract knowledge from source tasks via training source prompts, we need to consider this correlation among source tasks for better transfer to target tasks. We propose a Bayesian approach where we work with the posterior distribution of prompts across source tasks. We show extensive experimental results on the standard benchmark NLP tasks, where our Bayesian multi-task transfer learning approach outperforms the state-of-the-art methods in many settings.
  arXiv  Detail & Related papers  (2024-02-13T16:57:02Z)
• One-Shot Learning as Instruction Data Prospector for Large Language Models [108.81681547472138]
  textscNuggets uses one-shot learning to select high-quality instruction data from extensive datasets. We show that instruction tuning with the top 1% of examples curated by textscNuggets substantially outperforms conventional methods employing the entire dataset.
  arXiv  Detail & Related papers  (2023-12-16T03:33:12Z)
• IDEAL: Influence-Driven Selective Annotations Empower In-Context Learners in Large Language Models [66.32043210237768]
  This paper introduces an influence-driven selective annotation method. It aims to minimize annotation costs while improving the quality of in-context examples. Experiments confirm the superiority of the proposed method on various benchmarks.
  arXiv  Detail & Related papers  (2023-10-16T22:53:54Z)
• Automated Few-shot Classification with Instruction-Finetuned Language Models [76.69064714392165]
  We show that AuT-Few outperforms state-of-the-art few-shot learning methods. We also show that AuT-Few is the best ranking method across datasets on the RAFT few-shot benchmark.
  arXiv  Detail & Related papers  (2023-05-21T21:50:27Z)
• Actively Discovering New Slots for Task-oriented Conversation [19.815466126158785]
  We propose a general new slot task in an information extraction fashion to realize human-in-the-loop learning. We leverage existing language tools to extract value candidates where the corresponding labels are leveraged as weak supervision signals. We conduct extensive experiments on several public datasets and compare with a bunch of competitive baselines to demonstrate our method.
  arXiv  Detail & Related papers  (2023-05-06T13:33:33Z)
• Instance-wise Prompt Tuning for Pretrained Language Models [72.74916121511662]
  Instance-wise Prompt Tuning (IPT) is the first prompt learning paradigm that injects knowledge from the input data instances to the prompts. IPT significantly outperforms task-based prompt learning methods, and achieves comparable performance to conventional finetuning with only 0.5% - 1.5% of tuned parameters.
  arXiv  Detail & Related papers  (2022-06-04T10:08:50Z)
• Prompt Consistency for Zero-Shot Task Generalization [118.81196556175797]
  In this paper, we explore methods to utilize unlabeled data to improve zero-shot performance. Specifically, we take advantage of the fact that multiple prompts can be used to specify a single task, and propose to regularize prompt consistency. Our approach outperforms the state-of-the-art zero-shot learner, T0, on 9 out of 11 datasets across 4 NLP tasks by up to 10.6 absolute points in terms of accuracy.
  arXiv  Detail & Related papers  (2022-04-29T19:18:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.