RLPrompt: Optimizing Discrete Text Prompts With Reinforcement Learning
- URL: http://arxiv.org/abs/2205.12548v1
- Date: Wed, 25 May 2022 07:50:31 GMT
- Title: RLPrompt: Optimizing Discrete Text Prompts With Reinforcement Learning
- Authors: Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo,
Tianmin Shu, Meng Song, Eric P. Xing, Zhiting Hu
- Abstract summary: This paper proposes RLPrompt, an efficient discrete prompt optimization approach with reinforcement learning (RL)
RLPrompt is flexibly applicable to different types of LMs, such as masked gibberish (e.g., grammaBERT) and left-to-right models (e.g., GPTs)
Experiments on few-shot classification and unsupervised text style transfer show superior performance over a wide range of existing finetuning or prompting methods.
- Score: 84.75064077323098
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Prompting has shown impressive success in enabling large pretrained language
models (LMs) to perform diverse NLP tasks, especially when only few downstream
data are available. Automatically finding the optimal prompt for each task,
however, is challenging. Most existing work resorts to tuning soft prompt
(e.g., embeddings) which falls short of interpretability, reusability across
LMs, and applicability when gradients are not accessible. Discrete prompt, on
the other hand, is difficult to optimize, and is often created by "enumeration
(e.g., paraphrasing)-then-selection" heuristics that do not explore the prompt
space systematically. This paper proposes RLPrompt, an efficient discrete
prompt optimization approach with reinforcement learning (RL). RLPrompt
formulates a parameter-efficient policy network that generates the desired
discrete prompt after training with reward. To overcome the complexity and
stochasticity of reward signals by the large LM environment, we incorporate
effective reward stabilization that substantially enhances the training
efficiency. RLPrompt is flexibly applicable to different types of LMs, such as
masked (e.g., BERT) and left-to-right models (e.g., GPTs), for both
classification and generation tasks. Experiments on few-shot classification and
unsupervised text style transfer show superior performance over a wide range of
existing finetuning or prompting methods. Interestingly, the resulting
optimized prompts are often ungrammatical gibberish text; and surprisingly,
those gibberish prompts are transferrable between different LMs to retain
significant performance, indicating LM prompting may not follow human language
patterns.
Related papers
- Hard Prompts Made Interpretable: Sparse Entropy Regularization for Prompt Tuning with RL [29.01858866450715]
We present RLPrompt, which aims to find optimal prompt tokens leveraging soft Q-learning.
While the results show promise, we have observed that the prompts frequently appear unnatural, which impedes their interpretability.
We address this limitation by using sparse Tsallis entropy regularization, a principled approach to filtering out unlikely tokens from consideration.
arXiv Detail & Related papers (2024-07-20T03:10:19Z) - MAPO: Boosting Large Language Model Performance with Model-Adaptive Prompt Optimization [73.7779735046424]
We show that different prompts should be adapted to different Large Language Models (LLM) to enhance their capabilities across various downstream tasks in NLP.
We then propose a model-adaptive prompt (MAPO) method that optimize the original prompts for each specific LLM in downstream tasks.
arXiv Detail & Related papers (2024-07-04T18:39:59Z) - Efficient Prompting Methods for Large Language Models: A Survey [50.171011917404485]
Prompting has become a mainstream paradigm for adapting large language models (LLMs) to specific natural language processing tasks.
This approach brings the additional computational burden of model inference and human effort to guide and control the behavior of LLMs.
We present the basic concepts of prompting, review the advances for efficient prompting, and highlight future research directions.
arXiv Detail & Related papers (2024-04-01T12:19:08Z) - Intent-based Prompt Calibration: Enhancing prompt optimization with
synthetic boundary cases [2.6159111710501506]
We introduce a new method for automatic prompt engineering, using a calibration process that iteratively refines the prompt to the user intent.
We demonstrate the effectiveness of our method with respect to strong proprietary models on real-world tasks such as moderation and generation.
arXiv Detail & Related papers (2024-02-05T15:28:43Z) - Query-Dependent Prompt Evaluation and Optimization with Offline Inverse
RL [62.824464372594576]
We aim to enhance arithmetic reasoning ability of Large Language Models (LLMs) through zero-shot prompt optimization.
We identify a previously overlooked objective of query dependency in such optimization.
We introduce Prompt-OIRL, which harnesses offline inverse reinforcement learning to draw insights from offline prompting demonstration data.
arXiv Detail & Related papers (2023-09-13T01:12:52Z) - PromptBoosting: Black-Box Text Classification with Ten Forward Passes [61.38341243907045]
We describe PromptBoosting, a query-efficient procedure for building a text classifier from a neural language model (LM) without access to the LM's parameters, gradients, or hidden representations.
Experiments show that PromptBoosting achieves state-of-the-art performance in multiple black-box few-shot classification tasks, and matches or outperforms full fine-tuning in both few-shot and standard learning paradigms, while training 10x faster than existing black-box methods.
arXiv Detail & Related papers (2022-12-19T06:04:54Z) - Bayesian Prompt Learning for Image-Language Model Generalization [64.50204877434878]
We use the regularization ability of Bayesian methods to frame prompt learning as a variational inference problem.
Our approach regularizes the prompt space, reduces overfitting to the seen prompts and improves the prompt generalization on unseen prompts.
We demonstrate empirically on 15 benchmarks that Bayesian prompt learning provides an appropriate coverage of the prompt space.
arXiv Detail & Related papers (2022-10-05T17:05:56Z) - Making Pre-trained Language Models End-to-end Few-shot Learners with
Contrastive Prompt Tuning [41.15017636192417]
We present CP-Tuning, the first end-to-end Contrastive Prompt Tuning framework for fine-tuning Language Models.
It is integrated with the task-invariant continuous prompt encoding technique with fully trainable prompt parameters.
Experiments over a variety of language understanding tasks used in IR systems and different PLMs show that CP-Tuning outperforms state-of-the-art methods.
arXiv Detail & Related papers (2022-04-01T02:24:24Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.