Unraveling the Mechanics of Learning-Based Demonstration Selection for In-Context Learning

• URL: http://arxiv.org/abs/2406.11890v2
• Date: Tue, 15 Oct 2024 10:53:55 GMT
• Title: Unraveling the Mechanics of Learning-Based Demonstration Selection for In-Context Learning
• Authors: Hui Liu, Wenya Wang, Hao Sun, Chris Xing Tian, Chenqi Kong, Xin Dong, Haoliang Li,
• Abstract summary: We analyze the working mechanisms of the learning-based demonstration selection methods. We empirically identify two important factors related to similarity measurement. We introduce two effective yet simplified exemplar selection methods catering to task-agnostic and task-specific demands.
• Score: 43.356895599336504
• License:
• Abstract: Large Language Models (LLMs) have demonstrated impressive in-context learning (ICL) capabilities from few-shot demonstration exemplars. While recent learning-based demonstration selection methods have proven beneficial to ICL by choosing more useful exemplars, their underlying mechanisms are opaque, hindering efforts to address limitations such as high training costs and poor generalization across tasks. These methods generally assume the selection process captures similarities between the exemplar and the target instance, however, it remains unknown what kinds of similarities are captured and vital to performing ICL. To dive into this question, we analyze the working mechanisms of the learning-based demonstration selection methods and empirically identify two important factors related to similarity measurement: 1) The ability to integrate different levels of task-agnostic text similarities between the input of exemplars and test cases enhances generalization power across different tasks. 2) Incorporating task-specific labels when measuring the similarities significantly improves the performance on each specific task. We validate these two findings through extensive quantitative and qualitative analyses across ten datasets and various LLMs. Based on our findings, we introduce two effective yet simplified exemplar selection methods catering to task-agnostic and task-specific demands, eliminating the costly LLM inference overhead.

Related papers

• In-Context Transfer Learning: Demonstration Synthesis by Transferring Similar Tasks [93.46282380831339]
  In-context learning helps large language models adapt to various tasks by providing demonstrations of the target task. We propose In-Context Transfer Learning (ICTL), which synthesizes target task demonstrations by transferring labeled demonstrations from similar source tasks. Experiments on Super-NI show that ICTL outperforms synthesis from scratch by 2.0% on average.
  arXiv  Detail & Related papers  (2024-10-02T13:37:54Z)
• Unveiling In-Context Learning: A Coordinate System to Understand Its Working Mechanism [28.751003584429615]
  Large language models (LLMs) exhibit remarkable in-context learning capabilities. Recent research presents two conflicting views on ICL. We provide a Two-Dimensional Coordinate System that unifies both views into a systematic framework.
  arXiv  Detail & Related papers  (2024-07-24T05:26:52Z)
• C-ICL: Contrastive In-context Learning for Information Extraction [54.39470114243744]
  c-ICL is a novel few-shot technique that leverages both correct and incorrect sample constructions to create in-context learning demonstrations. Our experiments on various datasets indicate that c-ICL outperforms previous few-shot in-context learning methods.
  arXiv  Detail & Related papers  (2024-02-17T11:28:08Z)
• Revisiting Demonstration Selection Strategies in In-Context Learning [66.11652803887284]
  Large language models (LLMs) have shown an impressive ability to perform a wide range of tasks using in-context learning (ICL) In this work, we first revisit the factors contributing to this variance from both data and model aspects, and find that the choice of demonstration is both data- and model-dependent. We propose a data- and model-dependent demonstration selection method, textbfTopK + ConE, based on the assumption that textitthe performance of a demonstration positively correlates with its contribution to the model's understanding of the test samples.
  arXiv  Detail & Related papers  (2024-01-22T16:25:27Z)
• LLMs Learn Task Heuristics from Demonstrations: A Heuristic-Driven Prompting Strategy for Document-Level Event Argument Extraction [12.673710691468264]
  We introduce the Heuristic-Driven Link-of- Analogy (HD-LoA) prompting to address the challenge of example selection. Inspired by the analogical reasoning of human, we propose the link-of-analogy prompting, which enables LLMs to process new situations. Experiments show that our method outperforms existing prompting methods and few-shot supervised learning methods on document-level EAE datasets.
  arXiv  Detail & Related papers  (2023-11-11T12:05:01Z)
• Active Learning Principles for In-Context Learning with Large Language Models [65.09970281795769]
  This paper investigates how Active Learning algorithms can serve as effective demonstration selection methods for in-context learning. We show that in-context example selection through AL prioritizes high-quality examples that exhibit low uncertainty and bear similarity to the test examples.
  arXiv  Detail & Related papers  (2023-05-23T17:16:04Z)
• Iterative Forward Tuning Boosts In-Context Learning in Language Models [88.25013390669845]
  In this study, we introduce a novel two-stage framework to boost in-context learning in large language models (LLMs) Specifically, our framework delineates the ICL process into two distinct stages: Deep-Thinking and test stages. The Deep-Thinking stage incorporates a unique attention mechanism, i.e., iterative enhanced attention, which enables multiple rounds of information accumulation.
  arXiv  Detail & Related papers  (2023-05-22T13:18:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.