Exploring the Role of Diversity in Example Selection for In-Context Learning

• URL: http://arxiv.org/abs/2505.01842v2
• Date: Tue, 06 May 2025 14:51:28 GMT
• Title: Exploring the Role of Diversity in Example Selection for In-Context Learning
• Authors: Janak Kapuriya, Manit Kaushik, Debasis Ganguly, Sumit Bhatia,
• Abstract summary: In-Context Learning (ICL) has gained prominence due to its ability to perform tasks without requiring extensive training data and its robustness to noisy labels.<n> relying solely on similarity-based selection may introduce topical biases in the retrieved contexts, potentially leading to suboptimal downstream performance.<n>We posit that reranking the retrieved context to enhance topical diversity can improve downstream task performance.
• Score: 15.124618479338686
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In-Context Learning (ICL) has gained prominence due to its ability to perform tasks without requiring extensive training data and its robustness to noisy labels. A typical ICL workflow involves selecting localized examples relevant to a given input using sparse or dense embedding-based similarity functions. However, relying solely on similarity-based selection may introduce topical biases in the retrieved contexts, potentially leading to suboptimal downstream performance. We posit that reranking the retrieved context to enhance topical diversity can improve downstream task performance. To achieve this, we leverage maximum marginal relevance (MMR) which balances topical similarity with inter-example diversity. Our experimental results demonstrate that diversifying the selected examples leads to consistent improvements in downstream performance across various context sizes and similarity functions. The implementation of our approach is made available at https://github.com/janak11111/Diverse-ICL.

Related papers

• Stochastic Encodings for Active Feature Acquisition [100.47043816019888]
  Active Feature Acquisition is an instance-wise, sequential decision making problem.<n>The aim is to dynamically select which feature to measure based on current observations, independently for each test instance.<n>Common approaches either use Reinforcement Learning, which experiences training difficulties, or greedily maximize the conditional mutual information of the label and unobserved features, which makes myopic.<n>We introduce a latent variable model, trained in a supervised manner. Acquisitions are made by reasoning about the features across many possible unobserved realizations in a latent space.
  arXiv  Detail & Related papers  (2025-08-03T23:48:46Z)
• The Role of Diversity in In-Context Learning for Large Language Models [18.191350991951595]
  We investigate the role of diversity in in-context example selection through experiments across a range of tasks.<n>Experiments on Llama-3.1, Gemma-2, and Mistral-v0.3 families of models show that diversity-aware selection methods improve performance.<n>We introduce a theoretical framework that explains the benefits of incorporating diversity in in-context example selection.
  arXiv  Detail & Related papers  (2025-05-26T02:37:26Z)
• Bayesian Example Selection Improves In-Context Learning for Speech, Text, and Visual Modalities [15.931776592470895]
  Large language models (LLMs) can adapt to new tasks through in-context learning (ICL) This paper proposes a novel Bayesian in-Context example Selection method (ByCS) for ICL.
  arXiv  Detail & Related papers  (2024-04-23T03:42:48Z)
• ParaICL: Towards Parallel In-Context Learning [74.38022919598443]
  Large language models (LLMs) have become the norm in natural language processing.<n>Few-shot in-context learning (ICL) relies on the choice of few-shot demonstration examples.<n>We propose a novel method named parallel in-context learning (ParaICL)
  arXiv  Detail & Related papers  (2024-03-31T05:56:15Z)
• $Se^2$: Sequential Example Selection for In-Context Learning [83.17038582333716]
  Large language models (LLMs) for in-context learning (ICL) need to be activated by demonstration examples. Prior work has extensively explored the selection of examples for ICL, predominantly following the "select then organize" paradigm. In this paper, we formulate the problem as a $Se$quential $Se$lection problem and introduce $Se2$, a sequential-aware method.
  arXiv  Detail & Related papers  (2024-02-21T15:35:04Z)
• Copula for Instance-wise Feature Selection and Ranking [24.09326839818306]
  We propose to incorporate the Gaussian copula, a powerful mathematical technique for capturing correlations between variables, into the current feature selection framework. Experimental results on both synthetic and real datasets, in terms of performance comparison and interpretability, demonstrate that our method is capable of capturing meaningful correlations.
  arXiv  Detail & Related papers  (2023-08-01T13:45:04Z)
• Learning to Retrieve In-Context Examples for Large Language Models [69.9707552694766]
  Large language models (LLMs) have demonstrated their ability to learn in-context. The effectiveness of in-context learning is heavily reliant on the quality of the selected examples. We propose a novel framework to iteratively train dense retrievers that can identify high-quality in-context examples.
  arXiv  Detail & Related papers  (2023-07-14T05:23:08Z)
• Active Learning Principles for In-Context Learning with Large Language Models [65.09970281795769]
  This paper investigates how Active Learning algorithms can serve as effective demonstration selection methods for in-context learning. We show that in-context example selection through AL prioritizes high-quality examples that exhibit low uncertainty and bear similarity to the test examples.
  arXiv  Detail & Related papers  (2023-05-23T17:16:04Z)
• Finding Support Examples for In-Context Learning [73.90376920653507]
  We propose LENS, a fiLter-thEN-Search method to tackle this challenge in two stages. First we filter the dataset to obtain informative in-context examples individually. Then we propose diversity-guided example search which iteratively refines and evaluates the selected example permutations.
  arXiv  Detail & Related papers  (2023-02-27T06:32:45Z)
• Compositional Exemplars for In-context Learning [21.961094715261133]
  Large pretrained language models (LMs) have shown impressive In-Context Learning (ICL) ability. We propose CEIL (Compositional Exemplars for In-context Learning) to model the interaction between the given input and in-context examples. We validate CEIL on 12 classification and generation datasets from 7 distinct NLP tasks, including sentiment analysis, paraphrase detection, natural language inference, commonsense reasoning, open-domain question answering, code generation, and semantic parsing.
  arXiv  Detail & Related papers  (2023-02-11T14:02:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.