IDEAL: Influence-Driven Selective Annotations Empower In-Context Learners in Large Language Models

• URL: http://arxiv.org/abs/2310.10873v2
• Date: Sat, 20 Jan 2024 03:58:10 GMT
• Title: IDEAL: Influence-Driven Selective Annotations Empower In-Context Learners in Large Language Models
• Authors: Shaokun Zhang, Xiaobo Xia, Zhaoqing Wang, Ling-Hao Chen, Jiale Liu, Qingyun Wu, Tongliang Liu
• Abstract summary: This paper introduces an influence-driven selective annotation method. It aims to minimize annotation costs while improving the quality of in-context examples. Experiments confirm the superiority of the proposed method on various benchmarks.
• Score: 66.32043210237768
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In-context learning is a promising paradigm that utilizes in-context examples as prompts for the predictions of large language models. These prompts are crucial for achieving strong performance. However, since the prompts need to be sampled from a large volume of annotated examples, finding the right prompt may result in high annotation costs. To address this challenge, this paper introduces an influence-driven selective annotation method that aims to minimize annotation costs while improving the quality of in-context examples. The essence of our method is to select a pivotal subset from a large-scale unlabeled data pool to annotate for the subsequent sampling of prompts. Specifically, a directed graph is first constructed to represent unlabeled data. Afterward, the influence of candidate unlabeled subsets is quantified with a diffusion process. A simple yet effective greedy algorithm for unlabeled data selection is lastly introduced. It iteratively selects the data if it provides a maximum marginal gain with respect to quantified influence. Compared with previous efforts on selective annotations, our influence-driven method works in an end-to-end manner, avoids an intractable explicit balance between data diversity and representativeness, and enjoys theoretical support. Experiments confirm the superiority of the proposed method on various benchmarks, achieving better performance under lower time consumption during subset selection. The project page is available at https://skzhang1.github.io/IDEAL/.

Related papers

• Deep Active Learning with Manifold-preserving Trajectory Sampling [2.0717982775472206]
  Active learning (AL) is for optimizing the selection of unlabeled data for annotation (labeling) Existing deep AL methods arguably suffer from bias incurred by clabeled data, which takes a much lower percentage than unlabeled data in AL context. We propose a novel method, namely Manifold-Preserving Trajectory Sampling (MPTS), aiming to enforce the feature space learned from labeled data to represent a more accurate manifold.
  arXiv  Detail & Related papers  (2024-10-21T03:04:09Z)
• Sub-SA: Strengthen In-context Learning via Submodular Selective Annotation [4.846839863393725]
  We propose Sub-SA (Submodular Selective ), a sub-module-based selective annotation method. The aim of Sub-SA is to reduce annotation costs while improving the quality of in-context examples. We also propose RPR (Reward and Penalty Regularization) to better balance the diversity and representativeness of the unlabeled dataset.
  arXiv  Detail & Related papers  (2024-07-08T07:47:30Z)
• FastGAS: Fast Graph-based Annotation Selection for In-Context Learning [53.17606395275021]
  In-context learning (ICL) empowers large language models (LLMs) to tackle new tasks by using a series of training instances as prompts. Existing methods have proposed to select a subset of unlabeled examples for annotation. We propose a graph-based selection method, FastGAS, designed to efficiently identify high-quality instances.
  arXiv  Detail & Related papers  (2024-06-06T04:05:54Z)
• Prioritizing Informative Features and Examples for Deep Learning from Noisy Data [4.741012804505562]
  We propose a systemic framework that prioritizes informative features and examples to enhance each stage of the development process. We first propose an approach to extract only informative features that are inherent to solving a target task by using auxiliary out-of-distribution data. Next, we introduce an approach that prioritizes informative examples from unlabeled noisy data in order to reduce the labeling cost of active learning.
  arXiv  Detail & Related papers  (2024-02-27T07:15:35Z)
• Towards Free Data Selection with General-Purpose Models [71.92151210413374]
  A desirable data selection algorithm can efficiently choose the most informative samples to maximize the utility of limited annotation budgets. Current approaches, represented by active learning methods, typically follow a cumbersome pipeline that iterates the time-consuming model training and batch data selection repeatedly. FreeSel bypasses the heavy batch selection process, achieving a significant improvement in efficiency and being 530x faster than existing active learning methods.
  arXiv  Detail & Related papers  (2023-09-29T15:50:14Z)
• Selective Annotation Makes Language Models Better Few-Shot Learners [97.07544941620367]
  Large language models can perform in-context learning, where they learn a new task from a few task demonstrations. This work examines the implications of in-context learning for the creation of datasets for new natural language tasks. We propose an unsupervised, graph-based selective annotation method, voke-k, to select diverse, representative examples to annotate.
  arXiv  Detail & Related papers  (2022-09-05T14:01:15Z)
• Active Learning for Coreference Resolution using Discrete Annotation [76.36423696634584]
  We improve upon pairwise annotation for active learning in coreference resolution. We ask annotators to identify mention antecedents if a presented mention pair is deemed not coreferent. In experiments with existing benchmark coreference datasets, we show that the signal from this additional question leads to significant performance gains per human-annotation hour.
  arXiv  Detail & Related papers  (2020-04-28T17:17:11Z)
• Improving Multi-Turn Response Selection Models with Complementary Last-Utterance Selection by Instance Weighting [84.9716460244444]
  We consider utilizing the underlying correlation in the data resource itself to derive different kinds of supervision signals. We conduct extensive experiments in two public datasets and obtain significant improvement in both datasets.
  arXiv  Detail & Related papers  (2020-02-18T06:29:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.