PLR: Plackett-Luce for Reordering In-Context Learning Examples

• URL: http://arxiv.org/abs/2603.21373v1
• Date: Sun, 22 Mar 2026 19:15:59 GMT
• Title: PLR: Plackett-Luce for Reordering In-Context Learning Examples
• Abstract summary: In-context learning adapts large language models by conditioning on a small set of ICL examples.<n>PLR replaces discrete ordering search with learning a probability distribution over orderings with the Plackett-Luce model.<n>PLR consistently improves few-shot accuracy for $k in 4, 8, 16, 32$ examples.
• Score: 13.775690509818753
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In-context learning (ICL) adapts large language models by conditioning on a small set of ICL examples, avoiding costly parameter updates. Among other factors, performance is often highly sensitive to the ordering of the examples. However, exhaustive search over the $n!$ possible orderings is infeasible. Therefore more efficient ordering methods use model confidence measures (e.g., label-probability entropy) over label sets or take a direct approach to finding the best ordering. We propose PLR, a probabilistic approach to in-context example ordering that replaces discrete ordering search with learning a probability distribution over orderings with the Plackett-Luce model. PLR models orderings using a Plackett-Luce distribution and iteratively updates its parameters to concentrate probability mass on high-performing orderings under a task-level metric. Candidate orderings are sampled efficiently via a Gumbel perturb-and-sort procedure. Experiments on multiple classification benchmarks show that PLR consistently improves few-shot accuracy for $k \in \{4, 8, 16, 32\}$ examples, and we further demonstrate gains on mathematical reasoning tasks where label-based ordering methods are not applicable. Our code is available at https://github.com/Batorskq/PLR.

Related papers

• Towards Order Fairness: Mitigating LLMs Order Sensitivity through Dual Group Advantage Optimization [20.259122922188126]
  Large Language Models (LLMs) suffer from order bias, where their performance is affected by the arrangement order of input elements.<n>textbfDGAO aims to improve model accuracy and order stability simultaneously.
  arXiv  Detail & Related papers  (2026-05-12T11:31:18Z)
• Order Matters: Rethinking Prompt Construction in In-Context Learning [52.19217980839306]
  In-context learning (ICL) enables large language models to perform new tasks by conditioning on a sequence of examples.<n>Most prior work assumes that which examples are chosen has a far greater effect on performance than how those examples are ordered.<n>We revisit this assumption and conduct a systematic comparison between the effect of selection and ordering.
  arXiv  Detail & Related papers  (2025-11-12T19:57:55Z)
• GRAPE: Let GPRO Supervise Query Rewriting by Ranking for Retrieval [19.73916326078242]
  The CLIP model has become a cornerstone of large-scale retrieval systems by aligning text and image data in a unified embedding space.<n>To avoid costly retraining, existing methods mainly adopt query-rewriting strategies with large language models (LLMs)<n>We address this challenge with GRAPE, a plug-and-play enhancement approach that incorporates ranking signals into retrieval-guided query rewriting.
  arXiv  Detail & Related papers  (2025-09-27T15:36:59Z)
• Order Independence With Finetuning [0.0]
  Large language models (LLMs) demonstrate remarkable performance on many NLP tasks, yet often exhibit order dependence.<n>Recent work proposes Set-Based Prompting (SBP) as a way to remove order information from designated token subsets.<n>We introduce a fine-tuning strategy that integrates SBP into the training process, "pulling" these set-formatted prompts closer to the model's training manifold.
  arXiv  Detail & Related papers  (2025-03-30T15:38:43Z)
• Multimodal Label Relevance Ranking via Reinforcement Learning [30.03543589748649]
  We introduce a novel method for multimodal label relevance ranking, named Label Relevance Ranking with Proximal Policy Optimization (LRtextsuperscript2PPO) LRtextsuperscript2PPO first utilizes partial order pairs in the target domain to train a reward model. We meticulously design state representation and a policy loss tailored for ranking tasks, enabling LRtextsuperscript2PPO to boost the performance of label relevance ranking model.
  arXiv  Detail & Related papers  (2024-07-18T07:06:49Z)
• Prompt Optimization with EASE? Efficient Ordering-aware Automated Selection of Exemplars [66.823588073584]
  Large language models (LLMs) have shown impressive capabilities in real-world applications. The quality of these exemplars in the prompt greatly impacts performance. Existing methods fail to adequately account for the impact of exemplar ordering on the performance.
  arXiv  Detail & Related papers  (2024-05-25T08:23:05Z)
• In-Context Example Ordering Guided by Label Distributions [34.30216341226014]
  We formulate in-context example ordering as an optimization problem. Inspired by the idea of learning from label proportions, we propose two principles for in-context example ordering guided by model's probability predictions. We demonstrate our approach outperforms the baselines by improving the classification accuracy, reducing model miscalibration, and also by selecting better in-context examples.
  arXiv  Detail & Related papers  (2024-02-18T04:08:10Z)
• Instruction Position Matters in Sequence Generation with Large Language Models [67.87516654892343]
  Large language models (LLMs) are capable of performing conditional sequence generation tasks, such as translation or summarization. We propose enhancing the instruction-following capability of LLMs by shifting the position of task instructions after the input sentences.
  arXiv  Detail & Related papers  (2023-08-23T12:36:57Z)
• RetICL: Sequential Retrieval of In-Context Examples with Reinforcement Learning [53.52699766206808]
  We propose Retrieval for In-Context Learning (RetICL), a learnable method for modeling and optimally selecting examples sequentially for in-context learning. We evaluate RetICL on math word problem solving and scientific question answering tasks and show that it consistently outperforms or matches and learnable baselines.
  arXiv  Detail & Related papers  (2023-05-23T20:15:56Z)
• Relation Extraction as Open-book Examination: Retrieval-enhanced Prompt Tuning [109.7767515627765]
  We propose a new semiparametric paradigm of retrieval-enhanced prompt tuning for relation extraction. Our model infers relation through knowledge stored in the weights during training. Our method can achieve state-of-the-art in both standard supervised and few-shot settings.
  arXiv  Detail & Related papers  (2022-05-04T23:38:37Z)
• When in Doubt: Improving Classification Performance with Alternating Normalization [57.39356691967766]
  We introduce Classification with Alternating Normalization (CAN), a non-parametric post-processing step for classification. CAN improves classification accuracy for challenging examples by re-adjusting their predicted class probability distribution. We empirically demonstrate its effectiveness across a diverse set of classification tasks.
  arXiv  Detail & Related papers  (2021-09-28T02:55:42Z)
• Conditional Poisson Stochastic Beam Search [35.60062127942947]
  Conditional Poisson beam search (CPSBS) is a more natural alternative to Kool et. al. 2019's beam search (SBS) CPSBS produces lower variance and more efficient estimators than SBS, even showing improvements in high entropy settings.
  arXiv  Detail & Related papers  (2021-09-22T20:49:16Z)
• Pre-training Is (Almost) All You Need: An Application to Commonsense Reasoning [61.32992639292889]
  Fine-tuning of pre-trained transformer models has become the standard approach for solving common NLP tasks. We introduce a new scoring method that casts a plausibility ranking task in a full-text format. We show that our method provides a much more stable training phase across random restarts.
  arXiv  Detail & Related papers  (2020-04-29T10:54:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.