Investigation into In-Context Learning Capabilities of Transformers

• URL: http://arxiv.org/abs/2604.25858v1
• Date: Tue, 28 Apr 2026 16:57:55 GMT
• Title: Investigation into In-Context Learning Capabilities of Transformers
• Abstract summary: Transformers have demonstrated a strong ability for in-context learning (ICL)<n>We conduct a systematic empirical study of in-context learning for binary classification tasks.<n>Our results highlight the critical role of dimensionality, signal strength, and contextual information in determining when in-context learning succeeds and when it fails.
• Score: 14.14937947207076
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Transformers have demonstrated a strong ability for in-context learning (ICL), enabling models to solve previously unseen tasks using only example input output pairs provided at inference time. While prior theoretical work has established conditions under which transformers can perform linear classification in-context, the empirical scaling behavior governing when this mechanism succeeds remains insufficiently characterized. In this paper, we conduct a systematic empirical study of in-context learning for Gaussian-mixture binary classification tasks. Building on the theoretical framework of Frei and Vardi (2024), we analyze how in-context test accuracy depends on three fundamental factors: the input dimension, the number of in-context examples, and the number of pre-training tasks. Using a controlled synthetic setup and a linear in-context classifier formulation, we isolate the geometric conditions under which models successfully infer task structure from context alone. We additionally investigate the emergence of benign overfitting, where models memorize noisy in-context labels while still achieving strong generalization performance on clean test data. Through extensive sweeps across dimensionality, sequence length, task diversity, and signal-to-noise regimes, we identify the parameter regions in which this phenomenon arises and characterize how it depends on data geometry and training exposure. Our results provide a comprehensive empirical map of scaling behavior in in-context classification, highlighting the critical role of dimensionality, signal strength, and contextual information in determining when in-context learning succeeds and when it fails.

Related papers

• Invariant Learning Dynamics of Transformers in Inductive Reasoning Tasks [50.79340930913805]
  We study a class of inductive tasks that unifies several synthetic tasks known in the literature, including in-context n-grams and multi-hop reasoning.<n>By casting circuit formation as a low-dimensional dynamical phenomenon, we take a step toward a predictive theory of how Transformers learn.
  arXiv  Detail & Related papers  (2026-07-13T17:56:11Z)
• Causally Evaluating the Learnability of Formal Language Tasks [59.73001113068827]
  We rigorously investigate the relationship between data frequency and learnability.<n>We derive Kullback-Leibler divergence metrics to measure the learnability of specific sub-tasks.<n>Our experiments show that evaluating learnability without causal intervention leads to incorrect conclusions.
  arXiv  Detail & Related papers  (2026-06-08T17:58:36Z)
• Fine-Tuning Dynamics of In-Context Factual Recall in Transformers [63.1821972855273]
  In-context learning -- performing tasks based on examples given in the prompt -- has emerged in large language models.<n>We study how incontext learning leverages factual knowledge recall.<n>We analyze the supervised fine-tuning dynamics of a one-layer transformer on IC-recall data.
  arXiv  Detail & Related papers  (2026-05-26T23:47:26Z)
• A Framework for Quantifying How Pre-Training and Context Benefit In-Context Learning [52.07397258423034]
  We propose a new framework to analyze the ICL performance in a class of realistic settings.<n>We derive the precise relationship between ICL performance, context length and the KL divergence between pre-train and query task distribution.
  arXiv  Detail & Related papers  (2025-10-26T09:21:29Z)
• Next-token pretraining implies in-context learning [0.0]
  We show how models adapt to context when trained on token sequences, especially from non-ergodic sources.<n>Our information-theoretic framework precisely predicts these in-distribution ICL dynamics.<n>We also show that a model's in-context performance on any task is mathematically coupled to the ensemble of tasks seen in pretraining.
  arXiv  Detail & Related papers  (2025-05-23T21:00:18Z)
• In-Context Linear Regression Demystified: Training Dynamics and Mechanistic Interpretability of Multi-Head Softmax Attention [52.159541540613915]
  We study how multi-head softmax attention models are trained to perform in-context learning on linear data.<n>Our results reveal that in-context learning ability emerges from the trained transformer as an aggregated effect of its architecture and the underlying data distribution.
  arXiv  Detail & Related papers  (2025-03-17T02:00:49Z)
• Toward Understanding In-context vs. In-weight Learning [50.24035812301655]
  We identify simplified distributional properties that give rise to the emergence and disappearance of in-context learning.<n>We then extend the study to a full large language model, showing how fine-tuning on various collections of natural language prompts can elicit similar in-context and in-weight learning behaviour.
  arXiv  Detail & Related papers  (2024-10-30T14:09:00Z)
• Asymptotic theory of in-context learning by linear attention [37.3817914656799]
  In-context learning is a cornerstone of Transformers' success.<n>Questions about the necessary sample complexity, pretraining task diversity, and context length for successful ICL remain unresolved.
  arXiv  Detail & Related papers  (2024-05-20T03:24:24Z)
• The mechanistic basis of data dependence and abrupt learning in an in-context classification task [0.3626013617212666]
  We show that specific distributional properties inherent in language control the trade-off or simultaneous appearance of two forms of learning. In-context learning is driven by the abrupt emergence of an induction head, which subsequently competes with in-weights learning. We propose that the sharp transitions in attention-based networks arise due to a specific chain of multi-layer operations necessary to achieve ICL.
  arXiv  Detail & Related papers  (2023-12-03T20:53:41Z)
• A Theory of Emergent In-Context Learning as Implicit Structure Induction [8.17811111226145]
  Scaling large language models leads to an emergent capacity to learn in-context from example demonstrations. We argue that in-context learning relies on recombination of compositional operations found in natural language data. We show how in-context learning is supported by a representation of the input's compositional structure.
  arXiv  Detail & Related papers  (2023-03-14T15:24:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.