PromptWizard: Task-Aware Prompt Optimization Framework

• URL: http://arxiv.org/abs/2405.18369v2
• Date: Thu, 03 Oct 2024 09:45:47 GMT
• Title: PromptWizard: Task-Aware Prompt Optimization Framework
• Authors: Eshaan Agarwal, Joykirat Singh, Vivek Dani, Raghav Magazine, Tanuja Ganu, Akshay Nambi,
• Abstract summary: Large language models (LLMs) have transformed AI across diverse domains. Manual prompt engineering is both labor-intensive and domain-specific. We introduce PromptWizard, a novel, fully automated framework for discrete prompt optimization.
• Score: 2.618253052454435
• License:
• Abstract: Large language models (LLMs) have transformed AI across diverse domains, with prompting being central to their success in guiding model outputs. However, manual prompt engineering is both labor-intensive and domain-specific, necessitating the need for automated solutions. We introduce PromptWizard, a novel, fully automated framework for discrete prompt optimization, utilizing a self-evolving, self-adapting mechanism. Through a feedback-driven critique and synthesis process, PromptWizard achieves an effective balance between exploration and exploitation, iteratively refining both prompt instructions and in-context examples to generate human-readable, task-specific prompts. This guided approach systematically improves prompt quality, resulting in superior performance across 45 tasks. PromptWizard excels even with limited training data, smaller LLMs, and various LLM architectures. Additionally, our cost analysis reveals a substantial reduction in API calls, token usage, and overall cost, demonstrating PromptWizard's efficiency, scalability, and advantages over existing prompt optimization strategies.

Related papers

• AMPO: Automatic Multi-Branched Prompt Optimization [43.586044739174646]
  We present AMPO, an automatic prompt optimization method that can iteratively develop a multi-branched prompt using failure cases as feedback. In experiments across five tasks, AMPO consistently achieves the best results.
  arXiv  Detail & Related papers  (2024-10-11T10:34:28Z)
• StraGo: Harnessing Strategic Guidance for Prompt Optimization [35.96577924228001]
  StraGo is a novel approach designed to mitigate prompt drifting by leveraging insights from both successful and failed cases. It employs a how-to-do methodology, integrating in-context learning to formulate specific, actionable strategies. Experiments conducted across a range of tasks, including reasoning, natural language understanding, domain-specific knowledge, and industrial applications, demonstrate StraGo's superior performance.
  arXiv  Detail & Related papers  (2024-10-11T07:55:42Z)
• QPO: Query-dependent Prompt Optimization via Multi-Loop Offline Reinforcement Learning [58.767866109043055]
  We introduce Query-dependent Prompt Optimization (QPO), which iteratively fine-tune a small pretrained language model to generate optimal prompts tailored to the input queries. We derive insights from offline prompting demonstration data, which already exists in large quantities as a by-product of benchmarking diverse prompts on open-sourced tasks. Experiments on various LLM scales and diverse NLP and math tasks demonstrate the efficacy and cost-efficiency of our method in both zero-shot and few-shot scenarios.
  arXiv  Detail & Related papers  (2024-08-20T03:06:48Z)
• Efficient Prompting Methods for Large Language Models: A Survey [50.171011917404485]
  Prompting has become a mainstream paradigm for adapting large language models (LLMs) to specific natural language processing tasks. This approach brings the additional computational burden of model inference and human effort to guide and control the behavior of LLMs. We present the basic concepts of prompting, review the advances for efficient prompting, and highlight future research directions.
  arXiv  Detail & Related papers  (2024-04-01T12:19:08Z)
• Intent-based Prompt Calibration: Enhancing prompt optimization with synthetic boundary cases [2.6159111710501506]
  We introduce a new method for automatic prompt engineering, using a calibration process that iteratively refines the prompt to the user intent. We demonstrate the effectiveness of our method with respect to strong proprietary models on real-world tasks such as moderation and generation.
  arXiv  Detail & Related papers  (2024-02-05T15:28:43Z)
• Automatic Engineering of Long Prompts [79.66066613717703]
  Large language models (LLMs) have demonstrated remarkable capabilities in solving complex open-domain tasks. This paper investigates the performance of greedy algorithms and genetic algorithms for automatic long prompt engineering. Our results show that the proposed automatic long prompt engineering algorithm achieves an average of 9.2% accuracy gain on eight tasks in Big Bench Hard.
  arXiv  Detail & Related papers  (2023-11-16T07:42:46Z)
• Connecting Large Language Models with Evolutionary Algorithms Yields Powerful Prompt Optimizers [70.18534453485849]
  EvoPrompt is a framework for discrete prompt optimization. It borrows the idea of evolutionary algorithms (EAs) as they exhibit good performance and fast convergence. It significantly outperforms human-engineered prompts and existing methods for automatic prompt generation.
  arXiv  Detail & Related papers  (2023-09-15T16:50:09Z)
• OverPrompt: Enhancing ChatGPT through Efficient In-Context Learning [49.38867353135258]
  We propose OverPrompt, leveraging the in-context learning capability of LLMs to handle multiple task inputs. Our experiments show that OverPrompt can achieve cost-efficient zero-shot classification without causing significant detriment to task performance.
  arXiv  Detail & Related papers  (2023-05-24T10:08:04Z)
• RLPrompt: Optimizing Discrete Text Prompts With Reinforcement Learning [84.75064077323098]
  This paper proposes RLPrompt, an efficient discrete prompt optimization approach with reinforcement learning (RL) RLPrompt is flexibly applicable to different types of LMs, such as masked gibberish (e.g., grammaBERT) and left-to-right models (e.g., GPTs) Experiments on few-shot classification and unsupervised text style transfer show superior performance over a wide range of existing finetuning or prompting methods.
  arXiv  Detail & Related papers  (2022-05-25T07:50:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.