FastGAS: Fast Graph-based Annotation Selection for In-Context Learning

• URL: http://arxiv.org/abs/2406.03730v1
• Date: Thu, 6 Jun 2024 04:05:54 GMT
• Title: FastGAS: Fast Graph-based Annotation Selection for In-Context Learning
• Authors: Zihan Chen, Song Wang, Cong Shen, Jundong Li,
• Abstract summary: In-context learning (ICL) empowers large language models (LLMs) to tackle new tasks by using a series of training instances as prompts. Existing methods have proposed to select a subset of unlabeled examples for annotation. We propose a graph-based selection method, FastGAS, designed to efficiently identify high-quality instances.
• Score: 53.17606395275021
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In-context learning (ICL) empowers large language models (LLMs) to tackle new tasks by using a series of training instances as prompts. Since generating the prompts needs to sample from a vast pool of instances and annotate them (e.g., add labels in classification task), existing methods have proposed to select a subset of unlabeled examples for annotation, thus enhancing the quality of prompts and concurrently mitigating annotation costs. However, these methods often require a long time to select instances due to their complexity, hindering their practical viability. To address this limitation, we propose a graph-based selection method, FastGAS, designed to efficiently identify high-quality instances while minimizing computational overhead. Initially, we construct a data similarity graph based on instance similarities. Subsequently, employing a graph partitioning algorithm, we partition the graph into pieces. Within each piece (i.e., subgraph), we adopt a greedy approach to pick the most representative nodes. By aggregating nodes from diverse pieces and annotating the corresponding instances, we identify a set of diverse and representative instances for ICL. Compared to prior approaches, our method not only exhibits superior performance on different tasks but also significantly reduces selection time. In addition, we demonstrate the efficacy of our approach in LLMs of larger sizes.

Related papers

• Reasoning Graph Enhanced Exemplars Retrieval for In-Context Learning [13.381974811214764]
  Reasoning Graph-enhanced Exemplar Retrieval(RGER) RGER uses graph kernel to select exemplars with semantic and structural similarity. The efficacy of RGER on math and logit reasoning tasks showcases its superiority over state-of-the-art retrieval-based approaches.
  arXiv  Detail & Related papers  (2024-09-17T12:58:29Z)
• Prompt Optimization with EASE? Efficient Ordering-aware Automated Selection of Exemplars [66.823588073584]
  Large language models (LLMs) have shown impressive capabilities in real-world applications. The quality of these exemplars in the prompt greatly impacts performance. Existing methods fail to adequately account for the impact of exemplar ordering on the performance.
  arXiv  Detail & Related papers  (2024-05-25T08:23:05Z)
• Decoupling Weighing and Selecting for Integrating Multiple Graph Pre-training Tasks [58.65410800008769]
  This paper proposes a novel instance-level framework for integrating multiple graph pre-training tasks, Weigh And Select (WAS) It first adaptively learns an optimal combination of tasks for each instance from a given task pool, based on which a customized instance-level task weighing strategy is learned. Experiments on 16 graph datasets across node-level and graph-level downstream tasks have demonstrated that WAS can achieve comparable performance to other leading counterparts.
  arXiv  Detail & Related papers  (2024-03-03T05:29:49Z)
• IDEAL: Influence-Driven Selective Annotations Empower In-Context Learners in Large Language Models [66.32043210237768]
  This paper introduces an influence-driven selective annotation method. It aims to minimize annotation costs while improving the quality of in-context examples. Experiments confirm the superiority of the proposed method on various benchmarks.
  arXiv  Detail & Related papers  (2023-10-16T22:53:54Z)
• DiffusAL: Coupling Active Learning with Graph Diffusion for Label-Efficient Node Classification [1.0602247913671219]
  We introduce a novel active graph learning approach called DiffusAL, showing significant robustness in diverse settings. Most of our calculations for acquisition and training can be pre-processed, making DiffusAL more efficient compared to approaches combining diverse selection criteria. Our experiments on various benchmark datasets show that, unlike previous methods, our approach significantly outperforms random selection in 100% of all datasets and labeling budgets tested.
  arXiv  Detail & Related papers  (2023-07-31T20:30:13Z)
• Comparing Text Representations: A Theory-Driven Approach [2.893558866535708]
  We adapt general tools from computational learning theory to fit the specific characteristics of text datasets. We present a method to evaluate the compatibility between representations and tasks. This method provides a calibrated, quantitative measure of the difficulty of a classification-based NLP task.
  arXiv  Detail & Related papers  (2021-09-15T17:48:19Z)
• Selecting Relevant Features from a Multi-domain Representation for Few-shot Classification [91.67977602992657]
  We propose a new strategy based on feature selection, which is both simpler and more effective than previous feature adaptation approaches. We show that a simple non-parametric classifier built on top of such features produces high accuracy and generalizes to domains never seen during training.
  arXiv  Detail & Related papers  (2020-03-20T15:44:17Z)
• Optimal Clustering from Noisy Binary Feedback [75.17453757892152]
  We study the problem of clustering a set of items from binary user feedback. We devise an algorithm with a minimal cluster recovery error rate. For adaptive selection, we develop an algorithm inspired by the derivation of the information-theoretical error lower bounds.
  arXiv  Detail & Related papers  (2019-10-14T09:18:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.