Monte Carlo Sampling for Analyzing In-Context Examples

• URL: http://arxiv.org/abs/2503.22002v1
• Date: Thu, 27 Mar 2025 21:37:40 GMT
• Title: Monte Carlo Sampling for Analyzing In-Context Examples
• Authors: Stephanie Schoch, Yangfeng Ji,
• Abstract summary: We develop a Monte Carlo sampling-based method to study the impact of number of examples while explicitly accounting for effects from order and selected examples.<n>We find that while performance is robust to ordering and number of examples, there is an unexpected performance degradation compared to random sampling.
• Score: 19.740652268957522
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Prior works have shown that in-context learning is brittle to presentation factors such as the order, number, and choice of selected examples. However, ablation-based guidance on selecting the number of examples may ignore the interplay between different presentation factors. In this work we develop a Monte Carlo sampling-based method to study the impact of number of examples while explicitly accounting for effects from order and selected examples. We find that previous guidance on how many in-context examples to select does not always generalize across different sets of selected examples and orderings, and whether one-shot settings outperform zero-shot settings is highly dependent on the selected example. Additionally, inspired by data valuation, we apply our sampling method to in-context example selection to select examples that perform well across different orderings. We find a negative result, that while performance is robust to ordering and number of examples, there is an unexpected performance degradation compared to random sampling.

Related papers

• Compact Example-Based Explanations for Language Models [1.8772057593980798]
  Training data influence estimation methods quantify the contribution of training documents to a model's output.<n>As humans cannot interpret thousands of documents, only a small subset of the training data can be presented as an explanation.<n>We propose a novel selection relevance score, a retraining-free metric that quantifies how useful a set of examples is for explaining a model's output.
  arXiv  Detail & Related papers  (2026-01-07T10:36:46Z)
• Order Matters: Rethinking Prompt Construction in In-Context Learning [52.19217980839306]
  In-context learning (ICL) enables large language models to perform new tasks by conditioning on a sequence of examples.<n>Most prior work assumes that which examples are chosen has a far greater effect on performance than how those examples are ordered.<n>We revisit this assumption and conduct a systematic comparison between the effect of selection and ordering.
  arXiv  Detail & Related papers  (2025-11-12T19:57:55Z)
• Failures Are the Stepping Stones to Success: Enhancing Few-Shot In-Context Learning by Leveraging Negative Samples [3.4511221986774516]
  Large Language Models exhibit powerful few-shot in-context learning (ICL) capabilities, but the performance is highly sensitive to provided examples.<n>Recent research has focused on retrieving corresponding examples for each input query.<n>We propose a novel method that utilizes Negative samples to better select Positive sample examples.
  arXiv  Detail & Related papers  (2025-07-31T03:06:27Z)
• Bayesian Example Selection Improves In-Context Learning for Speech, Text, and Visual Modalities [15.931776592470895]
  Large language models (LLMs) can adapt to new tasks through in-context learning (ICL) This paper proposes a novel Bayesian in-Context example Selection method (ByCS) for ICL.
  arXiv  Detail & Related papers  (2024-04-23T03:42:48Z)
• Designing Informative Metrics for Few-Shot Example Selection [14.961505860372492]
  We propose a complexity-based prompt selection approach for sequence tagging tasks. This approach avoids the training of a dedicated model for selection of examples. We use both sentence- and word-level metrics to match the complexity of examples to the (test) sentence being considered.
  arXiv  Detail & Related papers  (2024-03-06T17:11:38Z)
• Addressing Order Sensitivity of In-Context Demonstration Examples in Causal Language Models [18.03259038587496]
  In-context learning can be significantly influenced by the order of in-context demonstration examples. We introduce an unsupervised fine-tuning method, termed the Information-Augmented and Consistency-Enhanced approach. Our proposed method can reduce the sensitivity of CausalLMs to the order of in-context examples and exhibit robust generalizability.
  arXiv  Detail & Related papers  (2024-02-23T22:39:12Z)
• In-Context Demonstration Selection with Cross Entropy Difference [95.21947716378641]
  Large language models (LLMs) can use in-context demonstrations to improve performance on zero-shot tasks. We present a cross-entropy difference (CED) method for selecting in-context demonstrations.
  arXiv  Detail & Related papers  (2023-05-24T05:04:00Z)
• RetICL: Sequential Retrieval of In-Context Examples with Reinforcement Learning [53.52699766206808]
  We propose Retrieval for In-Context Learning (RetICL), a learnable method for modeling and optimally selecting examples sequentially for in-context learning. We evaluate RetICL on math word problem solving and scientific question answering tasks and show that it consistently outperforms or matches and learnable baselines.
  arXiv  Detail & Related papers  (2023-05-23T20:15:56Z)
• Finding Support Examples for In-Context Learning [73.90376920653507]
  We propose LENS, a fiLter-thEN-Search method to tackle this challenge in two stages. First we filter the dataset to obtain informative in-context examples individually. Then we propose diversity-guided example search which iteratively refines and evaluates the selected example permutations.
  arXiv  Detail & Related papers  (2023-02-27T06:32:45Z)
• Leveraging Importance Weights in Subset Selection [45.54597544672441]
  We present a subset selection algorithm designed to work with arbitrary model families in a practical batch setting. Our algorithm, IWeS, selects examples by importance sampling where the sampling probability assigned to each example is based on the entropy of models trained on previously selected batches.
  arXiv  Detail & Related papers  (2023-01-28T02:07:31Z)
• True Few-Shot Learning with Language Models [78.42578316883271]
  We evaluate the few-shot ability of LMs when held-out examples are unavailable. Our findings suggest that prior work significantly overestimated the true few-shot ability of LMs.
  arXiv  Detail & Related papers  (2021-05-24T17:55:51Z)
• K-Shot Contrastive Learning of Visual Features with Multiple Instance Augmentations [67.46036826589467]
  $K$-Shot Contrastive Learning is proposed to investigate sample variations within individual instances. It aims to combine the advantages of inter-instance discrimination by learning discriminative features to distinguish between different instances. Experiment results demonstrate the proposed $K$-shot contrastive learning achieves superior performances to the state-of-the-art unsupervised methods.
  arXiv  Detail & Related papers  (2020-07-27T04:56:41Z)
• On conditional versus marginal bias in multi-armed bandits [105.07190334523304]
  The bias of the sample means of the arms in multi-armed bandits is an important issue in adaptive data analysis. We characterize the sign of the conditional bias of monotone functions of the rewards, including the sample mean. Our results hold for arbitrary conditioning events and leverage natural monotonicity properties of the data collection policy.
  arXiv  Detail & Related papers  (2020-02-19T20:16:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.