ScatterShot: Interactive In-context Example Curation for Text Transformation

• URL: http://arxiv.org/abs/2302.07346v1
• Date: Tue, 14 Feb 2023 21:13:31 GMT
• Title: ScatterShot: Interactive In-context Example Curation for Text Transformation
• Authors: Tongshuang Wu, Hua Shen, Daniel S. Weld, Jeffrey Heer, Marco Tulio Ribeiro
• Abstract summary: We present ScatterShot, an interactive system for building high-quality demonstration sets for in-context learning. ScatterShot iteratively slices unlabeled data into task-specific patterns, samples informative inputs from underexplored or not-yet-saturated slices in an active learning manner. In a user study, ScatterShot greatly helps users in covering different patterns in the input space and labeling in-context examples more efficiently.
• Score: 44.9405895390925
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The in-context learning capabilities of LLMs like GPT-3 allow annotators to customize an LLM to their specific tasks with a small number of examples. However, users tend to include only the most obvious patterns when crafting examples, resulting in underspecified in-context functions that fall short on unseen cases. Further, it is hard to know when "enough" examples have been included even for known patterns. In this work, we present ScatterShot, an interactive system for building high-quality demonstration sets for in-context learning. ScatterShot iteratively slices unlabeled data into task-specific patterns, samples informative inputs from underexplored or not-yet-saturated slices in an active learning manner, and helps users label more efficiently with the help of an LLM and the current example set. In simulation studies on two text perturbation scenarios, ScatterShot sampling improves the resulting few-shot functions by 4-5 percentage points over random sampling, with less variance as more examples are added. In a user study, ScatterShot greatly helps users in covering different patterns in the input space and labeling in-context examples more efficiently, resulting in better in-context learning and less user effort.

Related papers

• Integrated Image-Text Based on Semi-supervised Learning for Small Sample Instance Segmentation [1.3157419797035321]
  The article proposes a novel small sample instance segmentation solution from the perspective of maximizing the utilization of existing information. First, it helps the model fully utilize unlabeled data by learning to generate pseudo labels, increasing the number of available samples. Second, by integrating the features of text and image, more accurate classification results can be obtained.
  arXiv  Detail & Related papers  (2024-10-21T14:44:08Z)
• Prompt Optimization with EASE? Efficient Ordering-aware Automated Selection of Exemplars [66.823588073584]
  Large language models (LLMs) have shown impressive capabilities in real-world applications. The quality of these exemplars in the prompt greatly impacts performance. Existing methods fail to adequately account for the impact of exemplar ordering on the performance.
  arXiv  Detail & Related papers  (2024-05-25T08:23:05Z)
• Enhancing In-Context Learning with Answer Feedback for Multi-Span Question Answering [9.158919909909146]
  In this paper, we propose a novel way of employing labeled data such as it informs LLM of some undesired output. Experiments on three multi-span question answering datasets and a keyphrase extraction dataset show that our new prompting strategy consistently improves LLM's in-context learning performance.
  arXiv  Detail & Related papers  (2023-06-07T15:20:24Z)
• EXnet: Efficient In-context Learning for Data-less Text classification [0.0]
  We present EXnet, a model specifically designed to perform in-context learning without limitations on the number of examples. We argue that in-context learning is an effective method to increase task accuracy, and providing examples facilitates cross-task generalization. With extensive experiments, we show that even our smallest model (15M parameters) generalizes to several unseen classification tasks and domains.
  arXiv  Detail & Related papers  (2023-05-24T01:40:57Z)
• RetICL: Sequential Retrieval of In-Context Examples with Reinforcement Learning [53.52699766206808]
  We propose Retrieval for In-Context Learning (RetICL), a learnable method for modeling and optimally selecting examples sequentially for in-context learning. We evaluate RetICL on math word problem solving and scientific question answering tasks and show that it consistently outperforms or matches and learnable baselines.
  arXiv  Detail & Related papers  (2023-05-23T20:15:56Z)
• Active Learning Principles for In-Context Learning with Large Language Models [65.09970281795769]
  This paper investigates how Active Learning algorithms can serve as effective demonstration selection methods for in-context learning. We show that in-context example selection through AL prioritizes high-quality examples that exhibit low uncertainty and bear similarity to the test examples.
  arXiv  Detail & Related papers  (2023-05-23T17:16:04Z)
• Improving Few-Shot Performance of Language Models via Nearest Neighbor Calibration [12.334422701057674]
  We propose a novel nearest-neighbor calibration framework for in-context learning. It is inspired by a phenomenon that the in-context learning paradigm produces incorrect labels when inferring training instances. Experiments on various few-shot text classification tasks demonstrate that our method significantly improves in-context learning.
  arXiv  Detail & Related papers  (2022-12-05T12:49:41Z)
• Learning to Imagine: Diversify Memory for Incremental Learning using Unlabeled Data [69.30452751012568]
  We develop a learnable feature generator to diversify exemplars by adaptively generating diverse counterparts of exemplars. We introduce semantic contrastive learning to enforce the generated samples to be semantic consistent with exemplars. Our method does not bring any extra inference cost and outperforms state-of-the-art methods on two benchmarks.
  arXiv  Detail & Related papers  (2022-04-19T15:15:18Z)
• Learning by Examples Based on Multi-level Optimization [12.317568257671427]
  We propose a novel learning approach called Learning By Examples (LBE) Our approach automatically retrieves a set of training examples that are similar to query examples and predicts labels for query examples by using class labels of the retrieved examples. We conduct extensive experiments on various benchmarks where the results demonstrate the effectiveness of our method on both supervised and few-shot learning.
  arXiv  Detail & Related papers  (2021-09-22T16:33:06Z)
• Contrastive Learning with Adversarial Examples [79.39156814887133]
  Contrastive learning (CL) is a popular technique for self-supervised learning (SSL) of visual representations. This paper introduces a new family of adversarial examples for constrastive learning and using these examples to define a new adversarial training algorithm for SSL, denoted as CLAE.
  arXiv  Detail & Related papers  (2020-10-22T20:45:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.