StablePrompt: Automatic Prompt Tuning using Reinforcement Learning for Large Language Models

• URL: http://arxiv.org/abs/2410.07652v1
• Date: Thu, 10 Oct 2024 06:35:51 GMT
• Title: StablePrompt: Automatic Prompt Tuning using Reinforcement Learning for Large Language Models
• Authors: Minchan Kwon, Gaeun Kim, Jongsuk Kim, Haeil Lee, Junmo Kim,
• Abstract summary: Reinforcement Learning (RL) is widely used for prompt tuning, but its inherent instability and environmental dependency make it difficult to use in practice. We propose StablePrompt, which strikes a balance between training stability and search space, mitigating the instability of RL and producing high-performance prompts.
• Score: 21.556184207901115
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Finding appropriate prompts for the specific task has become an important issue as the usage of Large Language Models (LLM) has expanded. Reinforcement Learning (RL) is widely used for prompt tuning, but its inherent instability and environmental dependency make it difficult to use in practice. In this paper, we propose StablePrompt, which strikes a balance between training stability and search space, mitigating the instability of RL and producing high-performance prompts. We formulate prompt tuning as an online RL problem between the agent and target LLM and introduce Adaptive Proximal Policy Optimization (APPO). APPO introduces an LLM anchor model to adaptively adjust the rate of policy updates. This allows for flexible prompt search while preserving the linguistic ability of the pre-trained LLM. StablePrompt outperforms previous methods on various tasks including text classification, question answering, and text generation. Our code can be found in github.

Related papers

• Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models? [62.579951798437115]
  This work investigates iterative approximate evaluation for arbitrary prompts.<n>It introduces Model Predictive Prompt Selection (MoPPS), a Bayesian risk-predictive framework.<n>MoPPS reliably predicts prompt difficulty and accelerates training with significantly reduced rollouts.
  arXiv  Detail & Related papers  (2025-07-07T03:20:52Z)
• Prompt-Tuned LLM-Augmented DRL for Dynamic O-RAN Network Slicing [5.62872273155603]
  Large Language Models (LLMs) structure unorganized network feedback into meaningful latent representations.<n>In O-RAN slicing, concepts like SNR, power levels and throughput are semantically related.<n>We introduce a contextualization-based adaptation method that integrates learnable prompts into an LLM-augmented DRL framework.
  arXiv  Detail & Related papers  (2025-05-31T14:12:56Z)
• Reference Trustable Decoding: A Training-Free Augmentation Paradigm for Large Language Models [79.41139393080736]
  Large language models (LLMs) have rapidly advanced and demonstrated impressive capabilities. In-Context Learning (ICL) and. Efficient Fine-Tuning (PEFT) are currently two mainstream methods for augmenting. LLMs to downstream tasks. We propose Reference Trustable Decoding (RTD), a paradigm that allows models to quickly adapt to new tasks without fine-tuning.
  arXiv  Detail & Related papers  (2024-09-30T10:48:20Z)
• Hard Prompts Made Interpretable: Sparse Entropy Regularization for Prompt Tuning with RL [29.01858866450715]
  We present RLPrompt, which aims to find optimal prompt tokens leveraging soft Q-learning. While the results show promise, we have observed that the prompts frequently appear unnatural, which impedes their interpretability. We address this limitation by using sparse Tsallis entropy regularization, a principled approach to filtering out unlikely tokens from consideration.
  arXiv  Detail & Related papers  (2024-07-20T03:10:19Z)
• Soft Prompting for Unlearning in Large Language Models [11.504012974208466]
  This work focuses on investigating machine unlearning for Large Language Models motivated by data protection regulations. We propose a framework textbfSoft textbfPrompting for textbfUntextbflearning (SPUL) We conduct a rigorous evaluation of the proposed method and our results indicate that SPUL can significantly improve the trade-off between utility and forgetting.
  arXiv  Detail & Related papers  (2024-06-17T19:11:40Z)
• Relative Preference Optimization: Enhancing LLM Alignment through Contrasting Responses across Identical and Diverse Prompts [95.09994361995389]
  Relative Preference Optimization (RPO) is designed to discern between more and less preferred responses derived from both identical and related prompts. RPO has demonstrated a superior ability to align large language models with user preferences and to improve their adaptability during the training process.
  arXiv  Detail & Related papers  (2024-02-12T22:47:57Z)
• A Prompt Learning Framework for Source Code Summarization [19.24919436211323]
  This paper proposes an effective prompt learning framework for code summarization called PromptCS. PromptCS trains a prompt agent that can generate continuous prompts to unleash the potential for large language models in code summarization.
  arXiv  Detail & Related papers  (2023-12-26T14:37:55Z)
• LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Models [56.25156596019168]
  This paper introduces the LMRL-Gym benchmark for evaluating multi-turn RL for large language models (LLMs) Our benchmark consists of 8 different language tasks, which require multiple rounds of language interaction and cover a range of tasks in open-ended dialogue and text games.
  arXiv  Detail & Related papers  (2023-11-30T03:59:31Z)
• Self-regulating Prompts: Foundational Model Adaptation without Forgetting [112.66832145320434]
  We introduce a self-regularization framework for prompting called PromptSRC. PromptSRC guides the prompts to optimize for both task-specific and task-agnostic general representations.
  arXiv  Detail & Related papers  (2023-07-13T17:59:35Z)
• RLPrompt: Optimizing Discrete Text Prompts With Reinforcement Learning [84.75064077323098]
  This paper proposes RLPrompt, an efficient discrete prompt optimization approach with reinforcement learning (RL) RLPrompt is flexibly applicable to different types of LMs, such as masked gibberish (e.g., grammaBERT) and left-to-right models (e.g., GPTs) Experiments on few-shot classification and unsupervised text style transfer show superior performance over a wide range of existing finetuning or prompting methods.
  arXiv  Detail & Related papers  (2022-05-25T07:50:31Z)
• AdaPrompt: Adaptive Model Training for Prompt-based NLP [77.12071707955889]
  We propose AdaPrompt, adaptively retrieving external data for continual pretraining of PLMs. Experimental results on five NLP benchmarks show that AdaPrompt can improve over standard PLMs in few-shot settings. In zero-shot settings, our method outperforms standard prompt-based methods by up to 26.35% relative error reduction.
  arXiv  Detail & Related papers  (2022-02-10T04:04:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.