Rethinking Label Consistency of In-Context Learning: An Implicit Transductive Label Propagation Perspective

• URL: http://arxiv.org/abs/2512.12175v1
• Date: Sat, 13 Dec 2025 04:41:31 GMT
• Title: Rethinking Label Consistency of In-Context Learning: An Implicit Transductive Label Propagation Perspective
• Authors: Haoyang Chen, Richong Zhang, Junfan Chen,
• Abstract summary: Large language models (LLMs) perform in-context learning (ICL) with minimal supervised examples.<n>Current approaches typically employ retrieval models to select the top-K most semantically similar examples as demonstrations.<n>We propose a data synthesis method, leveraging both semantic and label information, and use TopK sampling with Synthetic Data (TopK-SD) to acquire demonstrations with consistent labels.
• Score: 34.36815585602357
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) perform in-context learning (ICL) with minimal supervised examples, which benefits various natural language processing (NLP) tasks. One of the critical research focus is the selection of prompt demonstrations. Current approaches typically employ retrieval models to select the top-K most semantically similar examples as demonstrations. However, we argue that existing methods are limited since the label consistency is not guaranteed during demonstration selection. Our cognition derives from the Bayesian view of ICL and our rethinking of ICL from the transductive label propagation perspective. We treat ICL as a transductive learning method and incorporate latent concepts from Bayesian view and deduce that similar demonstrations guide the concepts of query, with consistent labels serving as estimates. Based on this understanding, we establish a label propagation framework to link label consistency with propagation error bounds. To model label consistency, we propose a data synthesis method, leveraging both semantic and label information, and use TopK sampling with Synthetic Data (TopK-SD) to acquire demonstrations with consistent labels. TopK-SD outperforms original TopK sampling on multiple benchmarks. Our work provides a new perspective for understanding the working mechanisms within ICL.

Related papers

• Learn to Select: Exploring Label Distribution Divergence for In-Context Demonstration Selection in Text Classification [9.105555204653275]
  In-context learning (ICL) for text classification has demonstrated impressive performance on large language models (LLMs)<n>We propose a two-stage demonstration selection method, TopK + Label Distribution Divergence (L2D)<n>This enables the selection of demonstrations that are not only semantically similar but also aligned in label distribution with the test input.
  arXiv  Detail & Related papers  (2025-11-10T08:04:14Z)
• On the Relationship Between the Choice of Representation and In-Context Learning [38.52385081212209]
  In-context learning (ICL) is the ability of a large language model to learn a new task from a few demonstrations presented as part of the context.<n>Past studies have attributed a large portion of the success of ICL to the way these in-context demonstrations are represented.<n>We study the interaction between these two aspects in ICL, representation and learning.
  arXiv  Detail & Related papers  (2025-10-09T15:55:28Z)
• LLMs are Better Than You Think: Label-Guided In-Context Learning for Named Entity Recognition [10.920384665824807]
  In-context learning (ICL) enables large language models to perform new tasks using only a few demonstrations.<n>Existing ICL methods typically rely on task-agnostic semantic similarity for demonstration retrieval.<n>We introduce DEER, a training-free ICL approach that enables LLMs to make more informed entity predictions.
  arXiv  Detail & Related papers  (2025-05-29T17:54:32Z)
• Ambiguity-Aware In-Context Learning with Large Language Models [27.20414960164616]
  In-context learning (ICL) i.e. showing LLMs task-specific demonstrations has led to downstream gains with no task-specific fine-tuning required. This study investigates how to select good demonstrations for ICL. We find that it is beneficial to not only choose semantically similar ICL demonstrations but also to choose those that help resolve the inherent label ambiguity surrounding the test example.
  arXiv  Detail & Related papers  (2023-09-14T17:48:34Z)
• Channel-Wise Contrastive Learning for Learning with Noisy Labels [60.46434734808148]
  We introduce channel-wise contrastive learning (CWCL) to distinguish authentic label information from noise. Unlike conventional instance-wise contrastive learning (IWCL), CWCL tends to yield more nuanced and resilient features aligned with the authentic labels. Our strategy is twofold: firstly, using CWCL to extract pertinent features to identify cleanly labeled samples, and secondly, progressively fine-tuning using these samples.
  arXiv  Detail & Related papers  (2023-08-14T06:04:50Z)
• Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learning [77.7070536959126]
  In-context learning (ICL) emerges as a promising capability of large language models (LLMs) In this paper, we investigate the working mechanism of ICL through an information flow lens. We introduce an anchor re-weighting method to improve ICL performance, a demonstration compression technique to expedite inference, and an analysis framework for diagnosing ICL errors in GPT2-XL.
  arXiv  Detail & Related papers  (2023-05-23T15:26:20Z)
• Contrastive Label Enhancement [13.628665406039609]
  We propose Contrastive Label Enhancement (ConLE) to generate high-level features by contrastive learning strategy. We leverage the obtained high-level features to gain label distributions through a welldesigned training strategy.
  arXiv  Detail & Related papers  (2023-05-16T14:53:07Z)
• Exploring Structured Semantic Prior for Multi Label Recognition with Incomplete Labels [60.675714333081466]
  Multi-label recognition (MLR) with incomplete labels is very challenging. Recent works strive to explore the image-to-label correspondence in the vision-language model, ie, CLIP, to compensate for insufficient annotations. We advocate remedying the deficiency of label supervision for the MLR with incomplete labels by deriving a structured semantic prior.
  arXiv  Detail & Related papers  (2023-03-23T12:39:20Z)
• Label Matching Semi-Supervised Object Detection [85.99282969977541]
  Semi-supervised object detection has made significant progress with the development of mean teacher driven self-training. Label mismatch problem is not yet fully explored in the previous works, leading to severe confirmation bias during self-training. We propose a simple yet effective LabelMatch framework from two different yet complementary perspectives.
  arXiv  Detail & Related papers  (2022-06-14T05:59:41Z)
• A Theory-Driven Self-Labeling Refinement Method for Contrastive Representation Learning [111.05365744744437]
  Unsupervised contrastive learning labels crops of the same image as positives, and other image crops as negatives. In this work, we first prove that for contrastive learning, inaccurate label assignment heavily impairs its generalization for semantic instance discrimination. Inspired by this theory, we propose a novel self-labeling refinement approach for contrastive learning.
  arXiv  Detail & Related papers  (2021-06-28T14:24:52Z)
• Visual Transformer for Task-aware Active Learning [49.903358393660724]
  We present a novel pipeline for pool-based Active Learning. Our method exploits accessible unlabelled examples during training to estimate their co-relation with the labelled examples. Visual Transformer models non-local visual concept dependency between labelled and unlabelled examples.
  arXiv  Detail & Related papers  (2021-06-07T17:13:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.