Do prompt positions really matter?

• URL: http://arxiv.org/abs/2305.14493v4
• Date: Fri, 28 Jun 2024 12:04:53 GMT
• Title: Do prompt positions really matter?
• Authors: Junyu Mao, Stuart E. Middleton, Mahesan Niranjan,
• Abstract summary: Our findings quantify the substantial impact prompt position has on model performance. We observe that the prompt positions used in prior studies are often sub-optimal, and this observation is consistent even in widely used instruction-tuned models.
• Score: 2.6379212441113213
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Prompt-based models have gathered a lot of attention from researchers due to their remarkable advancements in the fields of zero-shot and few-shot learning. Developing an effective prompt template plays a critical role. However, prior studies have mainly focused on prompt vocabulary searching or embedding initialization within a predefined template with the prompt position fixed. In this empirical study, we conduct the most comprehensive analysis to date of prompt position for diverse Natural Language Processing (NLP) tasks. Our findings quantify the substantial impact prompt position has on model performance. We observe that the prompt positions used in prior studies are often sub-optimal, and this observation is consistent even in widely used instruction-tuned models. These findings suggest prompt position optimisation as a valuable research direction to augment prompt engineering methodologies and prompt position-aware instruction tuning as a potential way to build more robust models in the future.

Related papers

• Attention Basin: Why Contextual Position Matters in Large Language Models [16.11590856103274]
  We show that models systematically assign higher attention to items at the beginning and end of a sequence, while neglecting those in the middle.<n>We introduce Attention-Driven Reranking (AttnRank), a framework that estimates a model's intrinsic positional attention preferences.<n>AttnRank is a model-agnostic, training-free, and plug-and-play method with minimal computational overhead.
  arXiv  Detail & Related papers  (2025-08-07T08:08:08Z)
• ORPP: Self-Optimizing Role-playing Prompts to Enhance Language Model Capabilities [64.24517317344959]
  High-quality prompts are crucial for eliciting outstanding performance from large language models on complex tasks.<n>We propose ORPP, a framework that enhances model performance by optimizing and generating role-playing prompts.<n>We show that ORPP not only matches but in most cases surpasses existing mainstream prompt optimization methods in terms of performance.
  arXiv  Detail & Related papers  (2025-06-03T05:51:35Z)
• Dancing with Critiques: Enhancing LLM Reasoning with Stepwise Natural Language Self-Critique [66.94905631175209]
  We propose a novel inference-time scaling approach -- stepwise natural language self-critique (PANEL) It employs self-generated natural language critiques as feedback to guide the step-level search process. This approach bypasses the need for task-specific verifiers and the associated training overhead.
  arXiv  Detail & Related papers  (2025-03-21T17:59:55Z)
• Eliminating Position Bias of Language Models: A Mechanistic Approach [119.34143323054143]
  Position bias has proven to be a prevalent issue of modern language models (LMs) Our mechanistic analysis attributes the position bias to two components employed in nearly all state-of-the-art LMs: causal attention and relative positional encodings. By eliminating position bias, models achieve better performance and reliability in downstream tasks, including LM-as-a-judge, retrieval-augmented QA, molecule generation, and math reasoning.
  arXiv  Detail & Related papers  (2024-07-01T09:06:57Z)
• Position-Aware Parameter Efficient Fine-Tuning Approach for Reducing Positional Bias in LLMs [18.832135309689736]
  Recent advances in large language models (LLMs) have enhanced their ability to process long input contexts. Recent studies show a positional bias in LLMs, demonstrating varying performance depending on the location of useful information. We develop a Position-Aware PAPEFT approach which is composed of a data augmentation technique and an efficient parameter adapter.
  arXiv  Detail & Related papers  (2024-04-01T19:04:17Z)
• Exploring Large Language Model for Graph Data Understanding in Online Job Recommendations [63.19448893196642]
  We present a novel framework that harnesses the rich contextual information and semantic representations provided by large language models to analyze behavior graphs. By leveraging this capability, our framework enables personalized and accurate job recommendations for individual users.
  arXiv  Detail & Related papers  (2023-07-10T11:29:41Z)
• On the Role of Attention in Prompt-tuning [90.97555030446563]
  We study prompt-tuning for one-layer attention architectures and study contextual mixture-models. We show that softmax-prompt-attention is provably more expressive than softmax-self-attention and linear-prompt-attention. We also provide experiments that verify our theoretical insights on real datasets and demonstrate how prompt-tuning enables the model to attend to context-relevant information.
  arXiv  Detail & Related papers  (2023-06-06T06:23:38Z)
• Fairness-guided Few-shot Prompting for Large Language Models [93.05624064699965]
  In-context learning can suffer from high instability due to variations in training examples, example order, and prompt formats. We introduce a metric to evaluate the predictive bias of a fixed prompt against labels or a given attributes. We propose a novel search strategy based on the greedy search to identify the near-optimal prompt for improving the performance of in-context learning.
  arXiv  Detail & Related papers  (2023-03-23T12:28:25Z)
• Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation with Large Language Models [116.25562358482962]
  State-of-the-art neural language models can be used to solve ad-hoc language tasks without the need for supervised training. PromptIDE allows users to experiment with prompt variations, visualize prompt performance, and iteratively optimize prompts.
  arXiv  Detail & Related papers  (2022-08-16T17:17:53Z)
• Probing via Prompting [71.7904179689271]
  This paper introduces a novel model-free approach to probing, by formulating probing as a prompting task. We conduct experiments on five probing tasks and show that our approach is comparable or better at extracting information than diagnostic probes. We then examine the usefulness of a specific linguistic property for pre-training by removing the heads that are essential to that property and evaluating the resulting model's performance on language modeling.
  arXiv  Detail & Related papers  (2022-07-04T22:14:40Z)
• Making Pre-trained Language Models Good Long-tailed Learners [14.63635884051461]
  We check the hypothesis that prompt-tuning is also a promising choice for long-tailed classification. The results demonstrate that prompt-tuning exactly makes pre-trained language models at least good long-tailed learners.
  arXiv  Detail & Related papers  (2022-05-11T13:03:55Z)
• Prompt-Learning for Fine-Grained Entity Typing [40.983849729537795]
  We investigate the application of prompt-learning on fine-grained entity typing in fully supervised, few-shot and zero-shot scenarios. We propose a self-supervised strategy that carries out distribution-level optimization in prompt-learning to automatically summarize the information of entity types.
  arXiv  Detail & Related papers  (2021-08-24T09:39:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.