Auto-Demo Prompting: Leveraging Generated Outputs as Demonstrations for Enhanced Batch Prompting

• URL: http://arxiv.org/abs/2410.01724v1
• Date: Wed, 2 Oct 2024 16:34:40 GMT
• Title: Auto-Demo Prompting: Leveraging Generated Outputs as Demonstrations for Enhanced Batch Prompting
• Authors: Longyu Feng, Mengze Hong, Chen Jason Zhang,
• Abstract summary: "Auto-Demo Prompting" is a novel approach that leverages the question-output pairs from earlier questions within a batch as demonstrations for subsequent answer inference. Our method effectively bridges the gap between batch prompting and few-shot prompting, enhancing performance with only a slight compromise in token usage.
• Score: 0.8238423959893132
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Batch prompting is a common technique in large language models (LLMs) used to process multiple inputs simultaneously, aiming to improve computational efficiency. However, as batch sizes increase, performance degradation often occurs due to the model's difficulty in handling lengthy context inputs. Existing methods that attempt to mitigate these issues rely solely on batch data arrangement and majority voting rather than improving the design of the batch prompt itself. In this paper, we address these limitations by proposing "Auto-Demo Prompting," a novel approach that leverages the question-output pairs from earlier questions within a batch as demonstrations for subsequent answer inference. We provide a formal theoretical analysis of how Auto-Demo Prompting functions within the autoregressive generation process of LLMs, illustrating how it utilizes prior outputs to optimize the model's internal representations. Our method effectively bridges the gap between batch prompting and few-shot prompting, enhancing performance with only a slight compromise in token usage. Experimental results across five NLP tasks demonstrate its effectiveness in mitigating performance degradation and occasionally outperforming single prompts. Furthermore, it opens new avenues for applying few-shot learning techniques, such as demonstration selection, within batch prompting, making it a robust solution for real-world applications.

Related papers

• Achieving More with Less: Additive Prompt Tuning for Rehearsal-Free Class-Incremental Learning [76.32953653161417]
  Class-incremental learning enables models to learn new classes progressively while preserving knowledge of previously learned ones. Recent advances in this field have shifted towards parameter-efficient fine-tuning techniques. We present a novel prompt-based approach that addresses the limitation of current approaches.
  arXiv  Detail & Related papers  (2025-03-11T02:27:37Z)
• Robustness-aware Automatic Prompt Optimization [45.43458098928881]
  We propose BATprompt, a novel method for prompt generation designed to withstand input perturbations. Inspired by adversarial training techniques, BATprompt demonstrates strong performance on a variety of perturbed tasks. We evaluate BATprompt on multiple datasets across both language understanding and generation tasks.
  arXiv  Detail & Related papers  (2024-12-24T06:05:08Z)
• Hyperband-based Bayesian Optimization for Black-box Prompt Selection [15.756224286651237]
  Optimal prompt selection is crucial for maximizing large language model (LLM) performance on downstream tasks. We introduce HbBoPs, a novel Hyperband-based Bayesian optimization method for black-box prompt selection. Our approach combines a structural-aware deep kernel Gaussian Process to model prompt performance with Hyperband as a multi-fidelity scheduler.
  arXiv  Detail & Related papers  (2024-12-10T14:42:51Z)
• BATON: Enhancing Batch-wise Inference Efficiency for Large Language Models via Dynamic Re-batching [4.610983384440473]
  We propose BATON, an efficient batch-wise LLM inference scheme by dynamically adjusting processing batch. Compared to the state-of-the-art solution Orca, BATON improves query processing by up to 1.75 times.
  arXiv  Detail & Related papers  (2024-10-24T12:53:39Z)
• QPO: Query-dependent Prompt Optimization via Multi-Loop Offline Reinforcement Learning [58.767866109043055]
  We introduce Query-dependent Prompt Optimization (QPO), which iteratively fine-tune a small pretrained language model to generate optimal prompts tailored to the input queries. We derive insights from offline prompting demonstration data, which already exists in large quantities as a by-product of benchmarking diverse prompts on open-sourced tasks. Experiments on various LLM scales and diverse NLP and math tasks demonstrate the efficacy and cost-efficiency of our method in both zero-shot and few-shot scenarios.
  arXiv  Detail & Related papers  (2024-08-20T03:06:48Z)
• Large Language Models Prompting With Episodic Memory [53.8690170372303]
  We propose PrOmpting with Episodic Memory (POEM), a novel prompt optimization technique that is simple, efficient, and demonstrates strong generalization capabilities. In the testing phase, we optimize the sequence of examples for each test query by selecting the sequence that yields the highest total rewards from the top-k most similar training examples in the episodic memory. Our results show that POEM outperforms recent techniques like TEMPERA and RLPrompt by over 5.3% in various text classification tasks.
  arXiv  Detail & Related papers  (2024-08-14T11:19:28Z)
• Task Facet Learning: A Structured Approach to Prompt Optimization [14.223730629357178]
  We propose an algorithm that learns multiple facets of a task from a set of training examples. The resulting algorithm, UniPrompt, consists of a generative model to generate initial candidates for each prompt section. Empirical evaluation on multiple datasets and a real-world task shows that prompts generated using UniPrompt obtain higher accuracy than human-tuned prompts.
  arXiv  Detail & Related papers  (2024-06-15T04:54:26Z)
• Prompt Optimization with EASE? Efficient Ordering-aware Automated Selection of Exemplars [66.823588073584]
  Large language models (LLMs) have shown impressive capabilities in real-world applications. The quality of these exemplars in the prompt greatly impacts performance. Existing methods fail to adequately account for the impact of exemplar ordering on the performance.
  arXiv  Detail & Related papers  (2024-05-25T08:23:05Z)
• Efficient Prompting Methods for Large Language Models: A Survey [50.171011917404485]
  Prompting has become a mainstream paradigm for adapting large language models (LLMs) to specific natural language processing tasks. This approach brings the additional computational burden of model inference and human effort to guide and control the behavior of LLMs. We present the basic concepts of prompting, review the advances for efficient prompting, and highlight future research directions.
  arXiv  Detail & Related papers  (2024-04-01T12:19:08Z)
• Intent-based Prompt Calibration: Enhancing prompt optimization with synthetic boundary cases [2.6159111710501506]
  We introduce a new method for automatic prompt engineering, using a calibration process that iteratively refines the prompt to the user intent. We demonstrate the effectiveness of our method with respect to strong proprietary models on real-world tasks such as moderation and generation.
  arXiv  Detail & Related papers  (2024-02-05T15:28:43Z)
• Instance-wise Prompt Tuning for Pretrained Language Models [72.74916121511662]
  Instance-wise Prompt Tuning (IPT) is the first prompt learning paradigm that injects knowledge from the input data instances to the prompts. IPT significantly outperforms task-based prompt learning methods, and achieves comparable performance to conventional finetuning with only 0.5% - 1.5% of tuned parameters.
  arXiv  Detail & Related papers  (2022-06-04T10:08:50Z)
• IDPG: An Instance-Dependent Prompt Generation Method [58.45110542003139]
  Prompt tuning is a new, efficient NLP transfer learning paradigm that adds a task-specific prompt in each input instance during the model training stage. We propose a conditional prompt generation method to generate prompts for each input instance.
  arXiv  Detail & Related papers  (2022-04-09T15:45:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.