Scaling Performance and Low-Resource Annotation with Many-Shot In-Context Learning for Named Entity Recognition

• URL: http://arxiv.org/abs/2606.21890v1
• Date: Sat, 20 Jun 2026 05:39:59 GMT
• Title: Scaling Performance and Low-Resource Annotation with Many-Shot In-Context Learning for Named Entity Recognition
• Abstract summary: In-context learning (ICL) with large language models (LLMs) has emerged as a powerful alternative to fine-tuning for Named Entity Recognition (NER)<n>We conduct a comprehensive investigation of many-shot ICL for NER and explore its effectiveness in annotating and refining data for low-resource NER tasks.<n>Our experiments demonstrate that: (1) scaling to hundreds of in-context examples enables LLMs to match or even surpass the performance of fully supervised BERT models; and (2) using about one hundred human-labeled examples as demonstrations, many-shot in-context annotation can generate high-quality labeled
• Score: 52.07624601400713
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In-context learning (ICL) with large language models (LLMs) has emerged as a powerful alternative to fine-tuning for Named Entity Recognition (NER), achieving strong performance with minimal annotation and no additional training. However, prior work has shown that despite their adaptability, LLMs still lag behind fully supervised models such as fine-tuned BERT in structured tasks like NER. While existing studies on ICL for NER have mainly explored few-shot settings, the potential of scaling to hundreds of demonstrations has not been thoroughly investigated. To address this gap, we conduct a comprehensive investigation of many-shot ICL for NER and further explore its effectiveness in annotating and refining data for low-resource NER tasks. Specifically, we evaluate various LLMs across multiple domains using hundreds of ICL examples and then assess the feasibility of using many-shot ICL as a data annotation framework. Our experiments demonstrate that: (1) scaling to hundreds of in-context examples enables LLMs to match or even surpass the performance of fully supervised BERT models; and (2) using about one hundred human-labeled examples as demonstrations, many-shot in-context annotation can generate high-quality labeled data, leading to approximately 10% absolute F1 improvement over existing state-of-the-art approaches when used to fine-tune BERT on low-resource NER.

Related papers

• LC-ICL: Label-Guided Contrastive In-Context Learning for Robust Information Extraction [5.542001452666451]
  In this paper, we present LC-ICL, a novel few-shot technique that leverages both correct and incorrect sample constructions to create in-context learning demonstrations.<n>Our experiments on various datasets indicate that LC-ICL outperforms previous few-shot in-context learning methods.
  arXiv  Detail & Related papers  (2026-06-28T14:01:52Z)
• LLMs are Better Than You Think: Label-Guided In-Context Learning for Named Entity Recognition [10.920384665824807]
  In-context learning (ICL) enables large language models to perform new tasks using only a few demonstrations.<n>Existing ICL methods typically rely on task-agnostic semantic similarity for demonstration retrieval.<n>We introduce DEER, a training-free ICL approach that enables LLMs to make more informed entity predictions.
  arXiv  Detail & Related papers  (2025-05-29T17:54:32Z)
• MAPLE: Many-Shot Adaptive Pseudo-Labeling for In-Context Learning [53.02571749383208]
  In-Context Learning (ICL) empowers Large Language Models (LLMs) to tackle diverse tasks by incorporating multiple input-output examples.<n>Many-Shot Adaptive Pseudo-LabEling (MAPLE) is a novel influence-based many-shot ICL framework that utilizes pseudo-labeled samples to compensate for the lack of label information.
  arXiv  Detail & Related papers  (2025-05-22T04:54:27Z)
• Scaling Laws for Many-Shot In-Context Learning with Self-Generated Annotations [37.62305582749307]
  We study in-context learning with self-generated examples using a framework analogous to traditional semi-supervised learning.<n>Within this framework, we propose a simple baseline that outperforms ground-truth ICL in zero-shot, few-shot, and many-shot settings.<n>We introduce IterPSD, an iterative annotation approach that integrates iterative refinement and curriculum pseudo-labeling techniques.
  arXiv  Detail & Related papers  (2025-03-04T23:52:49Z)
• Visual RAG: Expanding MLLM visual knowledge without fine-tuning [5.341192792319891]
  This paper introduces Visual RAG, that synergically combines the MLLMs capability to learn from the context, with a retrieval mechanism.<n>In this way, the resulting system is not limited to the knowledge extracted from the training data, but can be updated rapidly and easily without fine-tuning.<n>It greatly reduces the computational costs for improving the model image classification performance, and augments the model knowledge to new visual domains and tasks it was not trained for.
  arXiv  Detail & Related papers  (2025-01-18T17:43:05Z)
• PICLe: Pseudo-Annotations for In-Context Learning in Low-Resource Named Entity Detection [56.916656013563355]
  In-context learning (ICL) enables Large Language Models to perform tasks using few demonstrations.<n>We propose PICLe, a framework for in-context learning with noisy, pseudo-annotated demonstrations.<n>We evaluate PICLe on five biomedical NED datasets and show that, with zero human annotation, PICLe outperforms ICL in low-resource settings.
  arXiv  Detail & Related papers  (2024-12-16T16:09:35Z)
• On Many-Shot In-Context Learning for Long-Context Evaluation [10.500629810624769]
  This paper delves into long-context language model evaluation through many-shot ICL.<n>We develop metrics to categorize ICL tasks into two groups: similar-sample learning (SSL) and all-sample learning (ASL)<n>We find that while state-of-the-art models demonstrate good performance up to 64k tokens in SSL tasks, many models experience significant performance drops at only 16k tokens in ASL tasks.
  arXiv  Detail & Related papers  (2024-11-11T17:00:59Z)
• C-ICL: Contrastive In-context Learning for Information Extraction [54.39470114243744]
  c-ICL is a novel few-shot technique that leverages both correct and incorrect sample constructions to create in-context learning demonstrations. Our experiments on various datasets indicate that c-ICL outperforms previous few-shot in-context learning methods.
  arXiv  Detail & Related papers  (2024-02-17T11:28:08Z)
• In-Context Learning for Few-Shot Nested Named Entity Recognition [53.55310639969833]
  We introduce an effective and innovative ICL framework for the setting of few-shot nested NER. We improve the ICL prompt by devising a novel example demonstration selection mechanism, EnDe retriever. In EnDe retriever, we employ contrastive learning to perform three types of representation learning, in terms of semantic similarity, boundary similarity, and label similarity.
  arXiv  Detail & Related papers  (2024-02-02T06:57:53Z)
• Z-ICL: Zero-Shot In-Context Learning with Pseudo-Demonstrations [97.41375480696972]
  We introduce Z-ICL, a new zero-shot method that closes the gap by constructing pseudo-demonstrations for a given test input. evaluation on nine classification datasets shows that Z-ICL outperforms previous zero-shot methods by a significant margin.
  arXiv  Detail & Related papers  (2022-12-19T21:34:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.