Order Matters: Rethinking Prompt Construction in In-Context Learning

• URL: http://arxiv.org/abs/2511.09700v1
• Date: Fri, 14 Nov 2025 01:05:05 GMT
• Title: Order Matters: Rethinking Prompt Construction in In-Context Learning
• Authors: Warren Li, Yiqian Wang, Zihan Wang, Jingbo Shang,
• Abstract summary: In-context learning (ICL) enables large language models to perform new tasks by conditioning on a sequence of examples.<n>Most prior work assumes that which examples are chosen has a far greater effect on performance than how those examples are ordered.<n>We revisit this assumption and conduct a systematic comparison between the effect of selection and ordering.
• Score: 52.19217980839306
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In-context learning (ICL) enables large language models to perform new tasks by conditioning on a sequence of examples. Most prior work reasonably and intuitively assumes that which examples are chosen has a far greater effect on performance than how those examples are ordered, leading to a focus on example selection. We revisit this assumption and conduct a systematic comparison between the effect of selection and ordering. Through controlled experiments on both classification and generation tasks, using multiple open-source model families (0.5B to 27B parameters) and GPT-5, we find that the variance in performance due to different example orderings is comparable to that from using entirely different example sets. Furthermore, we show that strong orderings can be identified using only a development set, achieving performance close to an oracle that selects the best ordering based on test labels. Our findings highlight the equal and intertwined importance of example selection and ordering in prompt design, calling for a reexamination of the assumptions held in ICL.

Related papers

• Monte Carlo Sampling for Analyzing In-Context Examples [19.740652268957522]
  We develop a Monte Carlo sampling-based method to study the impact of number of examples while explicitly accounting for effects from order and selected examples.<n>We find that while performance is robust to ordering and number of examples, there is an unexpected performance degradation compared to random sampling.
  arXiv  Detail & Related papers  (2025-03-27T21:37:40Z)
• Prompt Optimization with EASE? Efficient Ordering-aware Automated Selection of Exemplars [66.823588073584]
  Large language models (LLMs) have shown impressive capabilities in real-world applications. The quality of these exemplars in the prompt greatly impacts performance. Existing methods fail to adequately account for the impact of exemplar ordering on the performance.
  arXiv  Detail & Related papers  (2024-05-25T08:23:05Z)
• Experimental Design for Active Transductive Inference in Large Language Models [18.2671641610825]
  We use active learning for adaptive prompt design and call it Active In-context Prompt Design (AIPD) We design the LLM prompt by adaptively choosing few-shot examples from a training set to optimize performance on a test set. We propose two algorithms, GO and SAL, which differ in how the few-shot examples are chosen.
  arXiv  Detail & Related papers  (2024-04-12T23:27:46Z)
• ParaICL: Towards Parallel In-Context Learning [74.38022919598443]
  Large language models (LLMs) have become the norm in natural language processing.<n>Few-shot in-context learning (ICL) relies on the choice of few-shot demonstration examples.<n>We propose a novel method named parallel in-context learning (ParaICL)
  arXiv  Detail & Related papers  (2024-03-31T05:56:15Z)
• In-Context Example Ordering Guided by Label Distributions [34.30216341226014]
  We formulate in-context example ordering as an optimization problem. Inspired by the idea of learning from label proportions, we propose two principles for in-context example ordering guided by model's probability predictions. We demonstrate our approach outperforms the baselines by improving the classification accuracy, reducing model miscalibration, and also by selecting better in-context examples.
  arXiv  Detail & Related papers  (2024-02-18T04:08:10Z)
• RetICL: Sequential Retrieval of In-Context Examples with Reinforcement Learning [53.52699766206808]
  We propose Retrieval for In-Context Learning (RetICL), a learnable method for modeling and optimally selecting examples sequentially for in-context learning. We evaluate RetICL on math word problem solving and scientific question answering tasks and show that it consistently outperforms or matches and learnable baselines.
  arXiv  Detail & Related papers  (2023-05-23T20:15:56Z)
• Active Learning Principles for In-Context Learning with Large Language Models [65.09970281795769]
  This paper investigates how Active Learning algorithms can serve as effective demonstration selection methods for in-context learning. We show that in-context example selection through AL prioritizes high-quality examples that exhibit low uncertainty and bear similarity to the test examples.
  arXiv  Detail & Related papers  (2023-05-23T17:16:04Z)
• Compositional Exemplars for In-context Learning [21.961094715261133]
  Large pretrained language models (LMs) have shown impressive In-Context Learning (ICL) ability. We propose CEIL (Compositional Exemplars for In-context Learning) to model the interaction between the given input and in-context examples. We validate CEIL on 12 classification and generation datasets from 7 distinct NLP tasks, including sentiment analysis, paraphrase detection, natural language inference, commonsense reasoning, open-domain question answering, code generation, and semantic parsing.
  arXiv  Detail & Related papers  (2023-02-11T14:02:08Z)
• K-Shot Contrastive Learning of Visual Features with Multiple Instance Augmentations [67.46036826589467]
  $K$-Shot Contrastive Learning is proposed to investigate sample variations within individual instances. It aims to combine the advantages of inter-instance discrimination by learning discriminative features to distinguish between different instances. Experiment results demonstrate the proposed $K$-shot contrastive learning achieves superior performances to the state-of-the-art unsupervised methods.
  arXiv  Detail & Related papers  (2020-07-27T04:56:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.