In-Context Learning for Text Classification with Many Labels
- URL: http://arxiv.org/abs/2309.10954v2
- Date: Wed, 6 Dec 2023 03:34:00 GMT
- Title: In-Context Learning for Text Classification with Many Labels
- Authors: Aristides Milios, Siva Reddy, Dzmitry Bahdanau
- Abstract summary: In-context learning (ICL) using large language models for tasks with many labels is challenging due to the limited context window.
We use a pre-trained dense retrieval model to bypass this limitation.
We analyze the performance across number of in-context examples and different model scales.
- Score: 34.87532045406169
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: In-context learning (ICL) using large language models for tasks with many
labels is challenging due to the limited context window, which makes it
difficult to fit a sufficient number of examples in the prompt. In this paper,
we use a pre-trained dense retrieval model to bypass this limitation, giving
the model only a partial view of the full label space for each inference call.
Testing with recent open-source LLMs (OPT, LLaMA), we set new state of the art
performance in few-shot settings for three common intent classification
datasets, with no finetuning. We also surpass fine-tuned performance on
fine-grained sentiment classification in certain cases. We analyze the
performance across number of in-context examples and different model scales,
showing that larger models are necessary to effectively and consistently make
use of larger context lengths for ICL. By running several ablations, we analyze
the model's use of: a) the similarity of the in-context examples to the current
input, b) the semantic content of the class names, and c) the correct
correspondence between examples and labels. We demonstrate that all three are
needed to varying degrees depending on the domain, contrary to certain recent
works.
Related papers
- Less is More: Making Smaller Language Models Competent Subgraph Retrievers for Multi-hop KGQA [51.3033125256716]
We model the subgraph retrieval task as a conditional generation task handled by small language models.
Our base generative subgraph retrieval model, consisting of only 220M parameters, competitive retrieval performance compared to state-of-the-art models.
Our largest 3B model, when plugged with an LLM reader, sets new SOTA end-to-end performance on both the WebQSP and CWQ benchmarks.
arXiv Detail & Related papers (2024-10-08T15:22:36Z) - FastGAS: Fast Graph-based Annotation Selection for In-Context Learning [53.17606395275021]
In-context learning (ICL) empowers large language models (LLMs) to tackle new tasks by using a series of training instances as prompts.
Existing methods have proposed to select a subset of unlabeled examples for annotation.
We propose a graph-based selection method, FastGAS, designed to efficiently identify high-quality instances.
arXiv Detail & Related papers (2024-06-06T04:05:54Z) - Language Models for Text Classification: Is In-Context Learning Enough? [54.869097980761595]
Recent foundational language models have shown state-of-the-art performance in many NLP tasks in zero- and few-shot settings.
An advantage of these models over more standard approaches is the ability to understand instructions written in natural language (prompts)
This makes them suitable for addressing text classification problems for domains with limited amounts of annotated instances.
arXiv Detail & Related papers (2024-03-26T12:47:39Z) - ArcheType: A Novel Framework for Open-Source Column Type Annotation using Large Language Models [24.867534196627222]
We introduce ArcheType, a simple, practical method for context sampling, prompt serialization, model querying, and label remapping.
We establish a new state-of-the-art performance on zero-shot CTA benchmarks.
arXiv Detail & Related papers (2023-10-27T15:31:22Z) - BYOC: Personalized Few-Shot Classification with Co-Authored Class
Descriptions [2.076173115539025]
We propose a novel approach to few-shot text classification using an LLM.
Rather than few-shot examples, the LLM is prompted with descriptions of the salient features of each class.
Examples, questions, and answers are summarized to form the classification prompt.
arXiv Detail & Related papers (2023-10-09T19:37:38Z) - AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators [98.11286353828525]
GPT-3.5 series models have demonstrated remarkable few-shot and zero-shot ability across various NLP tasks.
We propose AnnoLLM, which adopts a two-step approach, explain-then-annotate.
We build the first conversation-based information retrieval dataset employing AnnoLLM.
arXiv Detail & Related papers (2023-03-29T17:03:21Z) - Stabilized In-Context Learning with Pre-trained Language Models for Few
Shot Dialogue State Tracking [57.92608483099916]
Large pre-trained language models (PLMs) have shown impressive unaided performance across many NLP tasks.
For more complex tasks such as dialogue state tracking (DST), designing prompts that reliably convey the desired intent is nontrivial.
We introduce a saliency model to limit dialogue text length, allowing us to include more exemplars per query.
arXiv Detail & Related papers (2023-02-12T15:05:10Z) - Semantic-Oriented Unlabeled Priming for Large-Scale Language Models [12.074766935042588]
We introduce Semantic-Oriented Unlabeled Priming (SOUP), a method that classifies examples by retrieving semantically similar unlabeled examples.
We also propose bag-of-contexts priming, a new priming strategy that is more suitable for our setting and enables the usage of more examples than fit into the context window.
arXiv Detail & Related papers (2022-02-12T19:50:59Z) - UniT: Unified Knowledge Transfer for Any-shot Object Detection and
Segmentation [52.487469544343305]
Methods for object detection and segmentation rely on large scale instance-level annotations for training.
We propose an intuitive and unified semi-supervised model that is applicable to a range of supervision.
arXiv Detail & Related papers (2020-06-12T22:45:47Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.