Efficient Prompting Methods for Large Language Models: A Survey

• URL: http://arxiv.org/abs/2404.01077v1
• Date: Mon, 1 Apr 2024 12:19:08 GMT
• Title: Efficient Prompting Methods for Large Language Models: A Survey
• Authors: Kaiyan Chang, Songcheng Xu, Chenglong Wang, Yingfeng Luo, Tong Xiao, Jingbo Zhu,
• Abstract summary: Prompting has become a mainstream paradigm for adapting large language models (LLMs) to specific natural language processing tasks. This approach brings the additional computational burden of model inference and human effort to guide and control the behavior of LLMs. We present the basic concepts of prompting, review the advances for efficient prompting, and highlight future research directions.
• Score: 50.171011917404485
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Prompting has become a mainstream paradigm for adapting large language models (LLMs) to specific natural language processing tasks. While this approach opens the door to in-context learning of LLMs, it brings the additional computational burden of model inference and human effort of manual-designed prompts, particularly when using lengthy and complex prompts to guide and control the behavior of LLMs. As a result, the LLM field has seen a remarkable surge in efficient prompting methods. In this paper, we present a comprehensive overview of these methods. At a high level, efficient prompting methods can broadly be categorized into two approaches: prompting with efficient computation and prompting with efficient design. The former involves various ways of compressing prompts, and the latter employs techniques for automatic prompt optimization. We present the basic concepts of prompting, review the advances for efficient prompting, and highlight future research directions.

Related papers

• A Sequential Optimal Learning Approach to Automated Prompt Engineering in Large Language Models [14.483240353801074]
  This paper proposes an optimal learning framework for automated prompt engineering. It is designed to sequentially identify effective prompt features while efficiently allocating a limited evaluation budget. Our framework provides a solution to deploying automated prompt engineering in a wider range applications.
  arXiv  Detail & Related papers  (2025-01-07T03:51:10Z)
• Generative Prompt Internalization [48.91617280112579]
  We propose Generative Prompt Internalization (GenPI), a lightweight method that employs a joint training approach. GenPI not only replicates the behavior of models with prompt inputs but also generates the content of the prompt. We demonstrate that our approach effectively internalizes complex prompts across various agent-based application scenarios.
  arXiv  Detail & Related papers  (2024-11-24T17:32:20Z)
• SpeechPrompt: Prompting Speech Language Models for Speech Processing Tasks [94.10497337235083]
  We are first to explore the potential of prompting speech LMs in the domain of speech processing. We reformulate speech processing tasks into speech-to-unit generation tasks. We show that the prompting method can achieve competitive performance compared to the strong fine-tuning method.
  arXiv  Detail & Related papers  (2024-08-23T13:00:10Z)
• A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications [11.568575664316143]
  This paper provides a structured overview of recent advancements in prompt engineering, categorized by application area. We provide a summary detailing the prompting methodology, its applications, the models involved, and the datasets utilized. This systematic analysis enables a better understanding of this rapidly developing field and facilitates future research by illuminating open challenges and opportunities for prompt engineering.
  arXiv  Detail & Related papers  (2024-02-05T19:49:13Z)
• MPrompt: Exploring Multi-level Prompt Tuning for Machine Reading Comprehension [19.12663587559988]
  We propose a multi-level prompt tuning (MPrompt) method for machine reading comprehension. It utilizes prompts at task-specific, domain-specific, and context-specific levels to enhance the comprehension of input semantics. We conducted extensive experiments on 12 benchmarks of various QA formats and achieved an average improvement of 1.94% over the state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-10-27T14:24:06Z)
• Bayesian Prompt Learning for Image-Language Model Generalization [64.50204877434878]
  We use the regularization ability of Bayesian methods to frame prompt learning as a variational inference problem. Our approach regularizes the prompt space, reduces overfitting to the seen prompts and improves the prompt generalization on unseen prompts. We demonstrate empirically on 15 benchmarks that Bayesian prompt learning provides an appropriate coverage of the prompt space.
  arXiv  Detail & Related papers  (2022-10-05T17:05:56Z)
• Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation with Large Language Models [116.25562358482962]
  State-of-the-art neural language models can be used to solve ad-hoc language tasks without the need for supervised training. PromptIDE allows users to experiment with prompt variations, visualize prompt performance, and iteratively optimize prompts.
  arXiv  Detail & Related papers  (2022-08-16T17:17:53Z)
• RLPrompt: Optimizing Discrete Text Prompts With Reinforcement Learning [84.75064077323098]
  This paper proposes RLPrompt, an efficient discrete prompt optimization approach with reinforcement learning (RL) RLPrompt is flexibly applicable to different types of LMs, such as masked gibberish (e.g., grammaBERT) and left-to-right models (e.g., GPTs) Experiments on few-shot classification and unsupervised text style transfer show superior performance over a wide range of existing finetuning or prompting methods.
  arXiv  Detail & Related papers  (2022-05-25T07:50:31Z)
• Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm [0.0]
  We discuss methods of prompt programming, emphasizing the usefulness of considering prompts through the lens of natural language. We introduce the idea of a metaprompt that seeds the model to generate its own natural language prompts for a range of tasks.
  arXiv  Detail & Related papers  (2021-02-15T05:27:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.