An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labels

• URL: http://arxiv.org/abs/2203.11364v1
• Date: Mon, 21 Mar 2022 21:51:43 GMT
• Title: An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labels
• Authors: Taylor Sorensen, Joshua Robinson, Christopher Michael Rytting, Alexander Glenn Shaw, Kyle Jeffrey Rogers, Alexia Pauline Delorey, Mahmoud Khalil, Nancy Fulda, David Wingate
• Abstract summary: We introduce a new method for selecting prompt templates textitwithout labeled examples and textitwithout direct access to the model. Across 8 datasets representing 7 distinct NLP tasks, we show that when a template has high mutual information, it also has high accuracy on the task.
• Score: 55.06990011183662
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Pre-trained language models derive substantial linguistic and factual knowledge from the massive corpora on which they are trained, and prompt engineering seeks to align these models to specific tasks. Unfortunately, existing prompt engineering methods require significant amounts of labeled data, access to model parameters, or both. We introduce a new method for selecting prompt templates \textit{without labeled examples} and \textit{without direct access to the model}. Specifically, over a set of candidate templates, we choose the template that maximizes the mutual information between the input and the corresponding model output. Across 8 datasets representing 7 distinct NLP tasks, we show that when a template has high mutual information, it also has high accuracy on the task. On the largest model, selecting prompts with our method gets 90\% of the way from the average prompt accuracy to the best prompt accuracy and requires no ground truth labels.

Related papers

• Detection and Measurement of Syntactic Templates in Generated Text [58.111650675717414]
  We offer an analysis of syntactic features to characterize general repetition in models. We find that models tend to produce templated text in downstream tasks at a higher rate than what is found in human-reference texts.
  arXiv  Detail & Related papers  (2024-06-28T19:34:23Z)
• A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications [11.568575664316143]
  This paper provides a structured overview of recent advancements in prompt engineering, categorized by application area. We provide a summary detailing the prompting methodology, its applications, the models involved, and the datasets utilized. This systematic analysis enables a better understanding of this rapidly developing field and facilitates future research by illuminating open challenges and opportunities for prompt engineering.
  arXiv  Detail & Related papers  (2024-02-05T19:49:13Z)
• Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvements [10.687101698324897]
  Large language models demonstrate a remarkable capability for learning to solve new tasks from a few examples. The prompt template, or the way the input examples are formatted to obtain the prompt, is an important yet often overlooked aspect of in-context learning. We show that a poor choice of the template can reduce the performance of the strongest models and inference methods to a random guess level.
  arXiv  Detail & Related papers  (2024-01-12T18:58:26Z)
• PromptNER: Prompt Locating and Typing for Named Entity Recognition [39.81221703760443]
  In this paper, we design a dual-slot multi-prompt template with the position slot and type slot to prompt locating and typing respectively. Multiple prompts can be input to the model simultaneously, and then the model extracts all entities by parallel predictions on the slots. Experimental results show that the proposed model achieves a significant performance improvement, especially in the cross-domain few-shot setting.
  arXiv  Detail & Related papers  (2023-05-26T17:16:11Z)
• Automated Few-shot Classification with Instruction-Finetuned Language Models [76.69064714392165]
  We show that AuT-Few outperforms state-of-the-art few-shot learning methods. We also show that AuT-Few is the best ranking method across datasets on the RAFT few-shot benchmark.
  arXiv  Detail & Related papers  (2023-05-21T21:50:27Z)
• Small Character Models Match Large Word Models for Autocomplete Under Memory Constraints [32.79377465262468]
  We study the more challenging open-domain setting consisting of low frequency user prompt patterns. Character-based representation is effective in reducing the overall model size. We show that a 20M parameter character model performs similar to an 80M parameter word model in the vanilla setting.
  arXiv  Detail & Related papers  (2022-10-06T23:29:59Z)
• Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation with Large Language Models [116.25562358482962]
  State-of-the-art neural language models can be used to solve ad-hoc language tasks without the need for supervised training. PromptIDE allows users to experiment with prompt variations, visualize prompt performance, and iteratively optimize prompts.
  arXiv  Detail & Related papers  (2022-08-16T17:17:53Z)
• Template-free Prompt Tuning for Few-shot NER [46.59447116255979]
  We propose a more elegant method to reformulate NER tasks as LM problems without any templates. Specifically, we discard the template construction process while maintaining the word prediction paradigm of pre-training models. Experimental results demonstrate the effectiveness of the proposed method over bert-tagger and template-based method under few-shot setting.
  arXiv  Detail & Related papers  (2021-09-28T07:19:24Z)
• When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data [84.87772675171412]
  We study the circumstances under which explanations of individual data points can improve modeling performance. We make use of three existing datasets with explanations: e-SNLI, TACRED, SemEval.
  arXiv  Detail & Related papers  (2021-02-03T18:57:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.