On the Relationship Between the Choice of Representation and In-Context Learning

• URL: http://arxiv.org/abs/2510.08372v1
• Date: Thu, 09 Oct 2025 15:55:28 GMT
• Title: On the Relationship Between the Choice of Representation and In-Context Learning
• Authors: Ioana Marinescu, Kyunghyun Cho, Eric Karl Oermann,
• Abstract summary: In-context learning (ICL) is the ability of a large language model to learn a new task from a few demonstrations presented as part of the context.<n>Past studies have attributed a large portion of the success of ICL to the way these in-context demonstrations are represented.<n>We study the interaction between these two aspects in ICL, representation and learning.
• Score: 38.52385081212209
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: In-context learning (ICL) is the ability of a large language model (LLM) to learn a new task from a few demonstrations presented as part of the context. Past studies have attributed a large portion of the success of ICL to the way these in-context demonstrations are represented, particularly to how labels are represented in classification tasks. On the other hand, observations of the learning capacity of ICL (i.e., the extent to which more in-context demonstrations can lead to higher performance) have been mixed, and ICL is often thought to occur only under specific conditions. The interaction between these two aspects in ICL, representation and learning, has not been studied in depth until now. We hypothesize that they are largely independent of one another, such that the representation of demonstrations determines the baseline accuracy of ICL, while learning from additional demonstrations improves only on top of this baseline. We validate this hypothesis by developing an optimization algorithm that can enumerate a spectrum of possible label sets (representations) varying in semantic relevance. We then perform ICL with varying numbers of in-context demonstrations for each of these label sets. We observed that learning happens regardless of the quality of the label set itself, although its efficiency, measured by the slope of improvement over in-context demonstrations, is conditioned on both the label set quality and the parameter count of the underlying language model. Despite the emergence of learning, the relative quality (accuracy) of the choice of a label set (representation) is largely maintained throughout learning, confirming our hypothesis and implying their orthogonality. Our work reveals a previously underexplored aspect of ICL: the independent effects of learning from demonstrations and their representations on ICL performance.

Related papers

• PICLe: Pseudo-Annotations for In-Context Learning in Low-Resource Named Entity Detection [56.916656013563355]
  In-context learning (ICL) enables Large Language Models to perform tasks using few demonstrations.<n>We propose PICLe, a framework for in-context learning with noisy, pseudo-annotated demonstrations.<n>We evaluate PICLe on five biomedical NED datasets and show that, with zero human annotation, PICLe outperforms ICL in low-resource settings.
  arXiv  Detail & Related papers  (2024-12-16T16:09:35Z)
• DemoShapley: Valuation of Demonstrations for In-Context Learning [20.26604061802236]
  Large language models (LLMs) using in-context learning (ICL) excel in many tasks without task-specific fine-tuning.<n>We propose DemoShapley and Beta-DemoShapley, inspired by Data Shapley and Beta Shapley, to assess the influence of individual demonstrations.
  arXiv  Detail & Related papers  (2024-10-10T01:35:03Z)
• Unveiling In-Context Learning: A Coordinate System to Understand Its Working Mechanism [28.751003584429615]
  Large language models (LLMs) exhibit remarkable in-context learning capabilities. Recent research presents two conflicting views on ICL. We provide a Two-Dimensional Coordinate System that unifies both views into a systematic framework.
  arXiv  Detail & Related papers  (2024-07-24T05:26:52Z)
• Does In-Context Learning Really Learn? Rethinking How Large Language Models Respond and Solve Tasks via In-Context Learning [41.606494950216764]
  In-context Learning (ICL) has emerged as a powerful capability alongside the development of scaled-up large language models (LLMs) This paper decomposes the overall performance of ICL into three dimensions, label space, format, and discrimination. We show that ICL exhibits significant efficacy in regulating the label space and format, which helps LLMs respond to desired label words.
  arXiv  Detail & Related papers  (2024-04-11T08:20:10Z)
• Revisiting Demonstration Selection Strategies in In-Context Learning [66.11652803887284]
  Large language models (LLMs) have shown an impressive ability to perform a wide range of tasks using in-context learning (ICL) In this work, we first revisit the factors contributing to this variance from both data and model aspects, and find that the choice of demonstration is both data- and model-dependent. We propose a data- and model-dependent demonstration selection method, textbfTopK + ConE, based on the assumption that textitthe performance of a demonstration positively correlates with its contribution to the model's understanding of the test samples.
  arXiv  Detail & Related papers  (2024-01-22T16:25:27Z)
• Comparable Demonstrations are Important in In-Context Learning: A Novel Perspective on Demonstration Selection [22.29452683679149]
  In-Context Learning (ICL) is an important paradigm for adapting Large Language Models (LLMs) to downstream tasks through a few demonstrations. This study explores the ICL mechanisms from a novel perspective, providing a deeper insight into the demonstration selection strategy for ICL.
  arXiv  Detail & Related papers  (2023-12-12T18:05:46Z)
• Ambiguity-Aware In-Context Learning with Large Language Models [27.20414960164616]
  In-context learning (ICL) i.e. showing LLMs task-specific demonstrations has led to downstream gains with no task-specific fine-tuning required. This study investigates how to select good demonstrations for ICL. We find that it is beneficial to not only choose semantically similar ICL demonstrations but also to choose those that help resolve the inherent label ambiguity surrounding the test example.
  arXiv  Detail & Related papers  (2023-09-14T17:48:34Z)
• Investigating the Learning Behaviour of In-context Learning: A Comparison with Supervised Learning [67.25698169440818]
  Large language models (LLMs) have shown remarkable capacity for in-context learning (ICL) We train the same LLMs with the same demonstration examples via ICL and supervised learning (SL), respectively, and investigate their performance under label perturbations. First, we find that gold labels have significant impacts on the downstream in-context performance, especially for large language models. Second, when comparing with SL, we show empirically that ICL is less sensitive to label perturbations than SL, and ICL gradually attains comparable performance to SL as the model size increases.
  arXiv  Detail & Related papers  (2023-07-28T09:03:19Z)
• Scaling In-Context Demonstrations with Structured Attention [75.41845145597875]
  We propose a better architectural design for in-context learning. Structured Attention for In-Context Learning replaces the full-attention by a structured attention mechanism. We show that SAICL achieves comparable or better performance than full attention while obtaining up to 3.4x inference speed-up.
  arXiv  Detail & Related papers  (2023-07-05T23:26:01Z)
• Non-Contrastive Learning Meets Language-Image Pre-Training [145.6671909437841]
  We study the validity of non-contrastive language-image pre-training (nCLIP) We introduce xCLIP, a multi-tasking framework combining CLIP and nCLIP, and show that nCLIP aids CLIP in enhancing feature semantics.
  arXiv  Detail & Related papers  (2022-10-17T17:57:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.