Prompt Valuation Based on Shapley Values

• URL: http://arxiv.org/abs/2312.15395v2
• Date: Mon, 16 Dec 2024 08:57:29 GMT
• Title: Prompt Valuation Based on Shapley Values
• Authors: Hanxi Liu, Xiaokai Mao, Haocheng Xia, Jian Lou, Jinfei Liu, Kui Ren,
• Abstract summary: Large language models (LLMs) excel on new tasks without additional training, simply by providing natural language prompts. In this paper, we utilize the Shapley value to fairly quantify the contributions of prompts. We validate the effectiveness of using the Shapley value for prompts as it effectively distinguishes and quantifies the contributions of each prompt.
• Score: 12.53456269310386
• License:
• Abstract: Large language models (LLMs) excel on new tasks without additional training, simply by providing natural language prompts that demonstrate how the task should be performed. Prompt ensemble methods comprehensively harness the knowledge of LLMs while mitigating individual biases and errors and further enhancing performance. However, more prompts do not necessarily lead to better results, and not all prompts are beneficial. A small number of high-quality prompts often outperform many low-quality prompts. Currently, there is a lack of a suitable method for evaluating the impact of prompts on the results. In this paper, we utilize the Shapley value to fairly quantify the contributions of prompts, helping to identify beneficial or detrimental prompts, and potentially guiding prompt valuation in data markets. Through extensive experiments employing various ensemble methods and utility functions on diverse tasks, we validate the effectiveness of using the Shapley value method for prompts as it effectively distinguishes and quantifies the contributions of each prompt.

Related papers

• The Prompt Alchemist: Automated LLM-Tailored Prompt Optimization for Test Case Generation [17.064672221710307]
  Large Language Models (LLMs) can generate useful test cases for given source code. The existing work primarily relies on human-written plain prompts.
  arXiv  Detail & Related papers  (2025-01-02T16:30:05Z)
• QPO: Query-dependent Prompt Optimization via Multi-Loop Offline Reinforcement Learning [58.767866109043055]
  We introduce Query-dependent Prompt Optimization (QPO), which iteratively fine-tune a small pretrained language model to generate optimal prompts tailored to the input queries. We derive insights from offline prompting demonstration data, which already exists in large quantities as a by-product of benchmarking diverse prompts on open-sourced tasks. Experiments on various LLM scales and diverse NLP and math tasks demonstrate the efficacy and cost-efficiency of our method in both zero-shot and few-shot scenarios.
  arXiv  Detail & Related papers  (2024-08-20T03:06:48Z)
• Exploring Lottery Prompts for Pre-trained Language Models [46.66885465183664]
  We explore the instance-level prompt and their generalizability. We find that for every instance, there is almost always a lottery prompt that induces the correct prediction from the PLM. Some strong lottery prompts have high performance over the whole training set.
  arXiv  Detail & Related papers  (2023-05-31T02:17:04Z)
• OverPrompt: Enhancing ChatGPT through Efficient In-Context Learning [49.38867353135258]
  We propose OverPrompt, leveraging the in-context learning capability of LLMs to handle multiple task inputs. Our experiments show that OverPrompt can achieve cost-efficient zero-shot classification without causing significant detriment to task performance.
  arXiv  Detail & Related papers  (2023-05-24T10:08:04Z)
• Toward Human Readable Prompt Tuning: Kubrick's The Shining is a good movie, and a good prompt too? [84.91689960190054]
  Large language models can perform new tasks in a zero-shot fashion, given natural language prompts. It is underexplored what factors make the prompts effective, especially when the prompts are natural language.
  arXiv  Detail & Related papers  (2022-12-20T18:47:13Z)
• Demystifying Prompts in Language Models via Perplexity Estimation [109.59105230163041]
  Performance of a prompt is coupled with the extent to which the model is familiar with the language it contains. We show that the lower the perplexity of the prompt is, the better the prompt is able to perform the task.
  arXiv  Detail & Related papers  (2022-12-08T02:21:47Z)
• RLPrompt: Optimizing Discrete Text Prompts With Reinforcement Learning [84.75064077323098]
  This paper proposes RLPrompt, an efficient discrete prompt optimization approach with reinforcement learning (RL) RLPrompt is flexibly applicable to different types of LMs, such as masked gibberish (e.g., grammaBERT) and left-to-right models (e.g., GPTs) Experiments on few-shot classification and unsupervised text style transfer show superior performance over a wide range of existing finetuning or prompting methods.
  arXiv  Detail & Related papers  (2022-05-25T07:50:31Z)
• Instance-aware Prompt Learning for Language Understanding and Generation [49.22899822734549]
  We propose an instance-aware prompt learning method that learns a different prompt for each instance. Our method achieves the state-of-the-art on the SuperGLUE few-shot learning benchmark.
  arXiv  Detail & Related papers  (2022-01-18T17:03:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.