Adaptive Prompting: Ad-hoc Prompt Composition for Social Bias Detection

• URL: http://arxiv.org/abs/2502.06487v1
• Date: Mon, 10 Feb 2025 14:06:19 GMT
• Title: Adaptive Prompting: Ad-hoc Prompt Composition for Social Bias Detection
• Authors: Maximilian Spliethöver, Tim Knebler, Fabian Fumagalli, Maximilian Muschalik, Barbara Hammer, Eyke Hüllermeier, Henning Wachsmuth,
• Abstract summary: We propose an adaptive prompting approach that predicts the optimal prompt composition ad-hoc for a given input. We apply our approach to social bias detection, a highly context-dependent task that requires semantic understanding. Our approach robustly ensures high detection performance, and is best in several settings.
• Score: 30.836788377666
• License:
• Abstract: Recent advances on instruction fine-tuning have led to the development of various prompting techniques for large language models, such as explicit reasoning steps. However, the success of techniques depends on various parameters, such as the task, language model, and context provided. Finding an effective prompt is, therefore, often a trial-and-error process. Most existing approaches to automatic prompting aim to optimize individual techniques instead of compositions of techniques and their dependence on the input. To fill this gap, we propose an adaptive prompting approach that predicts the optimal prompt composition ad-hoc for a given input. We apply our approach to social bias detection, a highly context-dependent task that requires semantic understanding. We evaluate it with three large language models on three datasets, comparing compositions to individual techniques and other baselines. The results underline the importance of finding an effective prompt composition. Our approach robustly ensures high detection performance, and is best in several settings. Moreover, first experiments on other tasks support its generalizability.

Related papers

• Has My System Prompt Been Used? Large Language Model Prompt Membership Inference [56.20586932251531]
  We develop Prompt Detective, a statistical method to reliably determine whether a given system prompt was used by a third-party language model. Our work reveals that even minor changes in system prompts manifest in distinct response distributions, enabling us to verify prompt usage with statistical significance.
  arXiv  Detail & Related papers  (2025-02-14T08:00:42Z)
• A Sequential Optimal Learning Approach to Automated Prompt Engineering in Large Language Models [14.483240353801074]
  This paper proposes an optimal learning framework for automated prompt engineering. It is designed to sequentially identify effective prompt features while efficiently allocating a limited evaluation budget. Our framework provides a solution to deploying automated prompt engineering in a wider range applications.
  arXiv  Detail & Related papers  (2025-01-07T03:51:10Z)
• Likelihood as a Performance Gauge for Retrieval-Augmented Generation [78.28197013467157]
  We show that likelihoods serve as an effective gauge for language model performance. We propose two methods that use question likelihood as a gauge for selecting and constructing prompts that lead to better performance.
  arXiv  Detail & Related papers  (2024-11-12T13:14:09Z)
• Improving Subject-Driven Image Synthesis with Subject-Agnostic Guidance [62.15866177242207]
  We show that through constructing a subject-agnostic condition, one could obtain outputs consistent with both the given subject and input text prompts. Our approach is conceptually simple and requires only minimal code modifications, but leads to substantial quality improvements.
  arXiv  Detail & Related papers  (2024-05-02T15:03:41Z)
• Query-Dependent Prompt Evaluation and Optimization with Offline Inverse RL [62.824464372594576]
  We aim to enhance arithmetic reasoning ability of Large Language Models (LLMs) through zero-shot prompt optimization. We identify a previously overlooked objective of query dependency in such optimization. We introduce Prompt-OIRL, which harnesses offline inverse reinforcement learning to draw insights from offline prompting demonstration data.
  arXiv  Detail & Related papers  (2023-09-13T01:12:52Z)
• Syntax-aware Hybrid prompt model for Few-shot multi-modal sentiment analysis [0.7693465097015469]
  Multimodal Sentiment Analysis (MSA) has been a popular topic in natural language processing nowadays. It is practical to explore the method for few-shot sentiment analysis in cross-modalities.
  arXiv  Detail & Related papers  (2023-06-02T07:22:43Z)
• ConES: Concept Embedding Search for Parameter Efficient Tuning Large Vision Language Models [21.15548013842187]
  We propose a Concept Embedding Search (ConES) approach by optimizing prompt embeddings. By dropping the text encoder, we are able to significantly speed up the learning process. Our approach can beat the prompt tuning and textual inversion methods in a variety of downstream tasks.
  arXiv  Detail & Related papers  (2023-05-30T12:45:49Z)
• Multi-Prompt with Depth Partitioned Cross-Modal Learning [25.239388488952375]
  Partitioned Multi-modal Prompt (PMPO) is a multi-modal prompting technique that extends the soft prompt from a single learnable prompt to multiple prompts. Our method divides the visual encoder depths and connects learnable prompts to the separated visual depths, enabling different prompts to capture hierarchical contextual depths. We evaluate the effectiveness of our approach on three challenging tasks: new class generalization, cross-dataset evaluation, and domain generalization.
  arXiv  Detail & Related papers  (2023-05-10T14:54:29Z)
• Fairness-guided Few-shot Prompting for Large Language Models [93.05624064699965]
  In-context learning can suffer from high instability due to variations in training examples, example order, and prompt formats. We introduce a metric to evaluate the predictive bias of a fixed prompt against labels or a given attributes. We propose a novel search strategy based on the greedy search to identify the near-optimal prompt for improving the performance of in-context learning.
  arXiv  Detail & Related papers  (2023-03-23T12:28:25Z)
• TEMPERA: Test-Time Prompting via Reinforcement Learning [57.48657629588436]
  We propose Test-time Prompt Editing using Reinforcement learning (TEMPERA) In contrast to prior prompt generation methods, TEMPERA can efficiently leverage prior knowledge. Our method achieves 5.33x on average improvement in sample efficiency when compared to the traditional fine-tuning methods.
  arXiv  Detail & Related papers  (2022-11-21T22:38:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.