Sequential Correlations Change In-Context Learning: Effective Context Length and Architectural Mismatch

• URL: http://arxiv.org/abs/2607.03660v1
• Date: Sat, 04 Jul 2026 01:47:29 GMT
• Title: Sequential Correlations Change In-Context Learning: Effective Context Length and Architectural Mismatch
• Abstract summary: We present a solvable model based on linear attention and test our predictions on realistic transformer architectures.<n>We identify two distinct effects: First, when the query token is independent of the context, within-context correlations induce an effective context length.<n>Second, when the query is also correlated with its context, test error is reduced, particularly for softmax attention when compared to linear attention.
• Score: 34.70343709092236
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modern sequence models have a striking capacity for in-context learning (ICL); they can perform new tasks based only on examples given in the prompt. Understanding how this ability emerges requires theory that captures important properties of natural data. Linear regression has served as a useful sandbox for ICL theory, but existing work has largely focused on prompts with independent examples. In this work, we extend this setting to sequentially correlated data, a basic feature of real sequences. We present a solvable model based on linear attention and test our predictions on realistic transformer architectures. We identify two distinct effects: First, when the query token is independent of the context, within-context correlations induce an effective context length: correlated prompts behave like shorter i.i.d. prompts. Second, when the query is also correlated with its context, test error is reduced, particularly for softmax attention when compared to linear attention. These results suggest that correlated prompts alter not only the effective sample size of in-context learning, but also which attention architectures are best matched to the task.

Related papers

• Fine-Tuning Dynamics of In-Context Factual Recall in Transformers [63.1821972855273]
  In-context learning -- performing tasks based on examples given in the prompt -- has emerged in large language models.<n>We study how incontext learning leverages factual knowledge recall.<n>We analyze the supervised fine-tuning dynamics of a one-layer transformer on IC-recall data.
  arXiv  Detail & Related papers  (2026-05-26T23:47:26Z)
• Many-Shot CoT-ICL: Making In-Context Learning Truly Learn [58.439517684779936]
  In-context learning (ICL) adapts large language models to new tasks by conditioning on demonstrations in the prompt without parameter updates.<n>We study many-shot chain-of-thought in-context learning (CoT-ICL) for reasoning and show that standard many-shot rules do not transfer.
  arXiv  Detail & Related papers  (2026-05-13T13:30:12Z)
• Investigation into In-Context Learning Capabilities of Transformers [14.14937947207076]
  Transformers have demonstrated a strong ability for in-context learning (ICL)<n>We conduct a systematic empirical study of in-context learning for binary classification tasks.<n>Our results highlight the critical role of dimensionality, signal strength, and contextual information in determining when in-context learning succeeds and when it fails.
  arXiv  Detail & Related papers  (2026-04-28T16:57:55Z)
• A Framework for Quantifying How Pre-Training and Context Benefit In-Context Learning [52.07397258423034]
  We propose a new framework to analyze the ICL performance in a class of realistic settings.<n>We derive the precise relationship between ICL performance, context length and the KL divergence between pre-train and query task distribution.
  arXiv  Detail & Related papers  (2025-10-26T09:21:29Z)
• Can an LLM Induce a Graph? Investigating Memory Drift and Context Length [11.214847796972705]
  Recently proposed evaluation benchmarks aim to characterize the effective context length and the forgetting tendencies of large language models (LLMs)<n>We argue for evaluating these models on more complex reasoning tasks that require them to induce structured relational knowledge from the text.<n>Our findings reveal that LLMs begin to exhibit memory drift and contextual forgetting at much shorter effective lengths when tasked with this form of relational reasoning.
  arXiv  Detail & Related papers  (2025-10-04T01:56:07Z)
• Provable In-Context Learning of Nonlinear Regression with Transformers [66.99048542127768]
  In-context learning (ICL) is the ability to perform unseen tasks using task specific prompts without updating parameters.<n>Recent research has actively explored the training dynamics behind ICL, with much of the focus on relatively simple tasks.<n>This paper investigates more complex nonlinear regression tasks, aiming to uncover how transformers acquire in-context learning capabilities.
  arXiv  Detail & Related papers  (2025-07-28T00:09:28Z)
• Contextualize-then-Aggregate: Circuits for In-Context Learning in Gemma-2 2B [51.74607395697567]
  In-Context Learning (ICL) is an intriguing ability of large language models (LLMs)<n>We use causal interventions to identify information flow in Gemma-2 2B for five naturalistic ICL tasks.<n>We find that the model infers task information using a two-step strategy we call contextualize-then-aggregate.
  arXiv  Detail & Related papers  (2025-03-31T18:33:55Z)
• In-Context Linear Regression Demystified: Training Dynamics and Mechanistic Interpretability of Multi-Head Softmax Attention [52.159541540613915]
  We study how multi-head softmax attention models are trained to perform in-context learning on linear data.<n>Our results reveal that in-context learning ability emerges from the trained transformer as an aggregated effect of its architecture and the underlying data distribution.
  arXiv  Detail & Related papers  (2025-03-17T02:00:49Z)
• Compositional Exemplars for In-context Learning [21.961094715261133]
  Large pretrained language models (LMs) have shown impressive In-Context Learning (ICL) ability. We propose CEIL (Compositional Exemplars for In-context Learning) to model the interaction between the given input and in-context examples. We validate CEIL on 12 classification and generation datasets from 7 distinct NLP tasks, including sentiment analysis, paraphrase detection, natural language inference, commonsense reasoning, open-domain question answering, code generation, and semantic parsing.
  arXiv  Detail & Related papers  (2023-02-11T14:02:08Z)
• Embarrassingly Simple Performance Prediction for Abductive Natural Language Inference [10.536415845097661]
  We propose a method for predicting the performance of NLI models without fine-tuning them. We show that the accuracy of the cosine similarity approach correlates strongly with the accuracy of the classification approach with a Pearson correlation coefficient of 0.65. Our method can lead to significant time savings in the process of model selection.
  arXiv  Detail & Related papers  (2022-02-21T18:10:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.