Large Language Models Are Human-Level Prompt Engineers

• URL: http://arxiv.org/abs/2211.01910v1
• Date: Thu, 3 Nov 2022 15:43:03 GMT
• Title: Large Language Models Are Human-Level Prompt Engineers
• Authors: Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, Jimmy Ba
• Abstract summary: We propose Automatic Prompt Engineer for automatic instruction generation and selection. We show that APE-engineered prompts can be applied to steer models toward truthfulness and/or informativeness.
• Score: 31.98042013940282
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: By conditioning on natural language instructions, large language models (LLMs) have displayed impressive capabilities as general-purpose computers. However, task performance depends significantly on the quality of the prompt used to steer the model, and most effective prompts have been handcrafted by humans. Inspired by classical program synthesis and the human approach to prompt engineering, we propose Automatic Prompt Engineer (APE) for automatic instruction generation and selection. In our method, we treat the instruction as the "program," optimized by searching over a pool of instruction candidates proposed by an LLM in order to maximize a chosen score function. To evaluate the quality of the selected instruction, we evaluate the zero-shot performance of another LLM following the selected instruction. Experiments on 24 NLP tasks show that our automatically generated instructions outperform the prior LLM baseline by a large margin and achieve better or comparable performance to the instructions generated by human annotators on 19/24 tasks. We conduct extensive qualitative and quantitative analyses to explore the performance of APE. We show that APE-engineered prompts can be applied to steer models toward truthfulness and/or informativeness, as well as to improve few-shot learning performance by simply prepending them to standard in-context learning prompts. Please check out our webpage at https://sites.google.com/view/automatic-prompt-engineer.

Related papers

• Evaluating the Instruction-following Abilities of Language Models using Knowledge Tasks [4.945902994386117]
  We focus on developing a benchmark for instruction-following where it is easy to verify both task performance as well as instruction-following capabilities. We adapt existing knowledge benchmarks and augment them with instructions that are a) conditional on correctly answering the knowledge task or b) use the space of candidate options in multiple-choice knowledge-answering tasks. We find that even large-scale instruction-tuned LLMs fail to follow simple instructions in zero-shot settings.
  arXiv  Detail & Related papers  (2024-10-16T19:07:37Z)
• MAPO: Boosting Large Language Model Performance with Model-Adaptive Prompt Optimization [73.7779735046424]
  We show that different prompts should be adapted to different Large Language Models (LLM) to enhance their capabilities across various downstream tasks in NLP. We then propose a model-adaptive prompt (MAPO) method that optimize the original prompts for each specific LLM in downstream tasks.
  arXiv  Detail & Related papers  (2024-07-04T18:39:59Z)
• Prompt Optimization with EASE? Efficient Ordering-aware Automated Selection of Exemplars [66.823588073584]
  Large language models (LLMs) have shown impressive capabilities in real-world applications. The quality of these exemplars in the prompt greatly impacts performance. Existing methods fail to adequately account for the impact of exemplar ordering on the performance.
  arXiv  Detail & Related papers  (2024-05-25T08:23:05Z)
• Efficient Prompting Methods for Large Language Models: A Survey [50.171011917404485]
  Prompting has become a mainstream paradigm for adapting large language models (LLMs) to specific natural language processing tasks. This approach brings the additional computational burden of model inference and human effort to guide and control the behavior of LLMs. We present the basic concepts of prompting, review the advances for efficient prompting, and highlight future research directions.
  arXiv  Detail & Related papers  (2024-04-01T12:19:08Z)
• Intent-based Prompt Calibration: Enhancing prompt optimization with synthetic boundary cases [2.6159111710501506]
  We introduce a new method for automatic prompt engineering, using a calibration process that iteratively refines the prompt to the user intent. We demonstrate the effectiveness of our method with respect to strong proprietary models on real-world tasks such as moderation and generation.
  arXiv  Detail & Related papers  (2024-02-05T15:28:43Z)
• Auto-Instruct: Automatic Instruction Generation and Ranking for Black-Box Language Models [91.02730155418699]
  Large language models (LLMs) can perform a wide range of tasks by following natural language instructions. We introduce Auto-Instruct, a novel method to automatically improve the quality of instructions provided to LLMs. In experiments on 118 out-of-domain tasks, Auto-Instruct surpasses both human-written instructions and existing baselines of LLM-generated instructions.
  arXiv  Detail & Related papers  (2023-10-19T19:52:55Z)
• AutoHint: Automatic Prompt Optimization with Hint Generation [11.737818328656735]
  This paper presents AutoHint, a novel framework for automatic prompt engineering and optimization for Large Language Models (LLM) We propose a framework to inherit the merits of both in-context learning and zero-shot learning by incorporating enriched instructions derived from input-output demonstrations to optimize original prompt. We refer to the enrichment as the hint and propose a framework to automatically generate the hint from labeled data.
  arXiv  Detail & Related papers  (2023-07-13T00:49:27Z)
• TEMPERA: Test-Time Prompting via Reinforcement Learning [57.48657629588436]
  We propose Test-time Prompt Editing using Reinforcement learning (TEMPERA) In contrast to prior prompt generation methods, TEMPERA can efficiently leverage prior knowledge. Our method achieves 5.33x on average improvement in sample efficiency when compared to the traditional fine-tuning methods.
  arXiv  Detail & Related papers  (2022-11-21T22:38:20Z)
• AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts [46.03503882865222]
  AutoPrompt is an automated method to create prompts for a diverse set of tasks based on a gradient-guided search. We show that masked language models (MLMs) have an inherent capability to perform sentiment analysis and natural language inference without additional parameters or finetuning.
  arXiv  Detail & Related papers  (2020-10-29T22:54:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.