The Role of Diversity in In-Context Learning for Large Language Models

• URL: http://arxiv.org/abs/2505.19426v2
• Date: Thu, 05 Jun 2025 08:20:31 GMT
• Title: The Role of Diversity in In-Context Learning for Large Language Models
• Authors: Wenyang Xiao, Haoyu Zhao, Lingxiao Huang,
• Abstract summary: We investigate the role of diversity in in-context example selection through experiments across a range of tasks.<n>Experiments on Llama-3.1, Gemma-2, and Mistral-v0.3 families of models show that diversity-aware selection methods improve performance.<n>We introduce a theoretical framework that explains the benefits of incorporating diversity in in-context example selection.
• Score: 18.191350991951595
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In-context learning (ICL) is a crucial capability of current large language models (LLMs), where the selection of examples plays a key role in performance. While most existing approaches focus on selecting the most similar examples to the query, the impact of diversity in example selection remains underexplored. We systematically investigate the role of diversity in in-context example selection through experiments across a range of tasks, from sentiment classification to more challenging math and code problems. Experiments on Llama-3.1, Gemma-2, and Mistral-v0.3 families of models show that diversity-aware selection methods improve performance, particularly on complex tasks like math and code, and enhance robustness to out-of-distribution queries. To support these findings, we introduce a theoretical framework that explains the benefits of incorporating diversity in in-context example selection.

Related papers

• Exploring the Role of Diversity in Example Selection for In-Context Learning [15.124618479338686]
  In-Context Learning (ICL) has gained prominence due to its ability to perform tasks without requiring extensive training data and its robustness to noisy labels.<n> relying solely on similarity-based selection may introduce topical biases in the retrieved contexts, potentially leading to suboptimal downstream performance.<n>We posit that reranking the retrieved context to enhance topical diversity can improve downstream task performance.
  arXiv  Detail & Related papers  (2025-05-03T15:13:58Z)
• Enhancing Contrastive Demonstration Selection with Semantic Diversity for Robust In-Context Machine Translation [0.0]
  We propose DiverseConE, a novel approach for demonstration selection in in-context learning for machine translation.<n>Our method builds upon contrastive selection by incorporating a diversity enhancement step based on embedding space dissimilarity.<n>Our results demonstrate that DiverseConE consistently outperforms strong baseline methods, including random selection, BM25, TopK, and a state-of-the-art contrastive selection method.
  arXiv  Detail & Related papers  (2025-04-12T18:35:04Z)
• Balanced Multi-Factor In-Context Learning for Multilingual Large Language Models [53.38288894305388]
  Multilingual large language models (MLLMs) are able to leverage in-context learning (ICL) to achieve high performance by leveraging cross-lingual knowledge transfer without parameter updates.<n>Three key factors influence multilingual ICL: (1) semantic similarity, (2) linguistic alignment, and (3) language-specific performance.<n>We propose balanced multi-factor ICL (textbfBMF-ICL), a method that quantifies and optimally balances these factors for improved example selection.
  arXiv  Detail & Related papers  (2025-02-17T06:56:33Z)
• Demonstration Selection for In-Context Learning via Reinforcement Learning [16.103533806505403]
  Relevance-Diversity Enhanced Selection (RDES) is an innovative approach to optimize the selection of diverse reference demonstrations.<n>RDES employs frameworks like Q-learning and a PPO-based variant to dynamically identify demonstrations that maximize diversity.<n>We demonstrate that RDES significantly enhances performance compared to ten established baselines.
  arXiv  Detail & Related papers  (2024-12-05T08:33:52Z)
• Unraveling the Mechanics of Learning-Based Demonstration Selection for In-Context Learning [43.356895599336504]
  We analyze the working mechanisms of the learning-based demonstration selection methods. We empirically identify two important factors related to similarity measurement. We introduce two effective yet simplified exemplar selection methods catering to task-agnostic and task-specific demands.
  arXiv  Detail & Related papers  (2024-06-14T03:34:02Z)
• $Se^2$: Sequential Example Selection for In-Context Learning [83.17038582333716]
  Large language models (LLMs) for in-context learning (ICL) need to be activated by demonstration examples. Prior work has extensively explored the selection of examples for ICL, predominantly following the "select then organize" paradigm. In this paper, we formulate the problem as a $Se$quential $Se$lection problem and introduce $Se2$, a sequential-aware method.
  arXiv  Detail & Related papers  (2024-02-21T15:35:04Z)
• Revisiting Demonstration Selection Strategies in In-Context Learning [66.11652803887284]
  Large language models (LLMs) have shown an impressive ability to perform a wide range of tasks using in-context learning (ICL) In this work, we first revisit the factors contributing to this variance from both data and model aspects, and find that the choice of demonstration is both data- and model-dependent. We propose a data- and model-dependent demonstration selection method, textbfTopK + ConE, based on the assumption that textitthe performance of a demonstration positively correlates with its contribution to the model's understanding of the test samples.
  arXiv  Detail & Related papers  (2024-01-22T16:25:27Z)
• Retrieval-augmented Multi-modal Chain-of-Thoughts Reasoning for Large Language Models [56.256069117502385]
  Chain of Thought (CoT) approaches can be used to enhance the capability of Large Language Models (LLMs) on complex reasoning tasks. However, the selection of optimal CoT demonstration examples in multi-modal reasoning remains less explored. We introduce a novel approach that addresses this challenge by using retrieval mechanisms to automatically select demonstration examples.
  arXiv  Detail & Related papers  (2023-12-04T08:07:21Z)
• RetICL: Sequential Retrieval of In-Context Examples with Reinforcement Learning [53.52699766206808]
  We propose Retrieval for In-Context Learning (RetICL), a learnable method for modeling and optimally selecting examples sequentially for in-context learning. We evaluate RetICL on math word problem solving and scientific question answering tasks and show that it consistently outperforms or matches and learnable baselines.
  arXiv  Detail & Related papers  (2023-05-23T20:15:56Z)
• Active Learning Principles for In-Context Learning with Large Language Models [65.09970281795769]
  This paper investigates how Active Learning algorithms can serve as effective demonstration selection methods for in-context learning. We show that in-context example selection through AL prioritizes high-quality examples that exhibit low uncertainty and bear similarity to the test examples.
  arXiv  Detail & Related papers  (2023-05-23T17:16:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.