Prompt Valuation Based on Shapley Values
- URL: http://arxiv.org/abs/2312.15395v1
- Date: Sun, 24 Dec 2023 03:37:11 GMT
- Title: Prompt Valuation Based on Shapley Values
- Authors: Hanxi Liu, Xiaokai Mao, Haocheng Xia, Jian Lou, Jinfei Liu
- Abstract summary: Large language models (LLMs) excel on new tasks without additional training, simply by providing natural language prompts.
In this paper, we utilize the Shapley value to fairly quantify the contributions of prompts.
We validate the effectiveness of using the Shapley value for prompts as it effectively distinguishes and quantifies the contributions of each prompt.
- Score: 5.072508764734943
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Large language models (LLMs) excel on new tasks without additional training,
simply by providing natural language prompts that demonstrate how the task
should be performed. Prompt ensemble methods comprehensively harness the
knowledge of LLMs while mitigating individual biases and errors and further
enhancing performance. However, more prompts do not necessarily lead to better
results, and not all prompts are beneficial. A small number of high-quality
prompts often outperform many low-quality prompts. Currently, there is a lack
of a suitable method for evaluating the impact of prompts on the results. In
this paper, we utilize the Shapley value to fairly quantify the contributions
of prompts, helping to identify beneficial or detrimental prompts, and
potentially guiding prompt valuation in data markets. Through extensive
experiments employing various ensemble methods and utility functions on diverse
tasks, we validate the effectiveness of using the Shapley value method for
prompts as it effectively distinguishes and quantifies the contributions of
each prompt.
Related papers
- QPO: Query-dependent Prompt Optimization via Multi-Loop Offline Reinforcement Learning [58.767866109043055]
We introduce Query-dependent Prompt Optimization (QPO), which iteratively fine-tune a small pretrained language model to generate optimal prompts tailored to the input queries.
We derive insights from offline prompting demonstration data, which already exists in large quantities as a by-product of benchmarking diverse prompts on open-sourced tasks.
Experiments on various LLM scales and diverse NLP and math tasks demonstrate the efficacy and cost-efficiency of our method in both zero-shot and few-shot scenarios.
arXiv Detail & Related papers (2024-08-20T03:06:48Z) - Efficient Prompting Methods for Large Language Models: A Survey [50.171011917404485]
Prompting has become a mainstream paradigm for adapting large language models (LLMs) to specific natural language processing tasks.
This approach brings the additional computational burden of model inference and human effort to guide and control the behavior of LLMs.
We present the basic concepts of prompting, review the advances for efficient prompting, and highlight future research directions.
arXiv Detail & Related papers (2024-04-01T12:19:08Z) - Exploring Lottery Prompts for Pre-trained Language Models [46.66885465183664]
We explore the instance-level prompt and their generalizability.
We find that for every instance, there is almost always a lottery prompt that induces the correct prediction from the PLM.
Some strong lottery prompts have high performance over the whole training set.
arXiv Detail & Related papers (2023-05-31T02:17:04Z) - Zero-shot Approach to Overcome Perturbation Sensitivity of Prompts [7.208567411886273]
Recent studies have demonstrated that natural-language prompts can help to leverage the knowledge learned by pre-trained language models for the binary sentence-level sentiment classification task.
This study aims to find high-quality prompts for the given task in a zero-shot setting.
We empirically demonstrate that the top-ranked prompts are high-quality and significantly outperform the base prompt and the prompts generated using few-shot learning for the binary sentence-level sentiment classification task.
arXiv Detail & Related papers (2023-05-25T03:36:43Z) - Toward Human Readable Prompt Tuning: Kubrick's The Shining is a good
movie, and a good prompt too? [84.91689960190054]
Large language models can perform new tasks in a zero-shot fashion, given natural language prompts.
It is underexplored what factors make the prompts effective, especially when the prompts are natural language.
arXiv Detail & Related papers (2022-12-20T18:47:13Z) - Demystifying Prompts in Language Models via Perplexity Estimation [109.59105230163041]
Performance of a prompt is coupled with the extent to which the model is familiar with the language it contains.
We show that the lower the perplexity of the prompt is, the better the prompt is able to perform the task.
arXiv Detail & Related papers (2022-12-08T02:21:47Z) - RLPrompt: Optimizing Discrete Text Prompts With Reinforcement Learning [84.75064077323098]
This paper proposes RLPrompt, an efficient discrete prompt optimization approach with reinforcement learning (RL)
RLPrompt is flexibly applicable to different types of LMs, such as masked gibberish (e.g., grammaBERT) and left-to-right models (e.g., GPTs)
Experiments on few-shot classification and unsupervised text style transfer show superior performance over a wide range of existing finetuning or prompting methods.
arXiv Detail & Related papers (2022-05-25T07:50:31Z) - Instance-aware Prompt Learning for Language Understanding and Generation [49.22899822734549]
We propose an instance-aware prompt learning method that learns a different prompt for each instance.
Our method achieves the state-of-the-art on the SuperGLUE few-shot learning benchmark.
arXiv Detail & Related papers (2022-01-18T17:03:25Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.