Demonstrations, CoT, and Prompting: A Theoretical Analysis of ICL

• URL: http://arxiv.org/abs/2603.19611v1
• Date: Fri, 20 Mar 2026 03:33:04 GMT
• Title: Demonstrations, CoT, and Prompting: A Theoretical Analysis of ICL
• Abstract summary: In-Context Learning (ICL) enables pretrained LLMs to adapt to downstream tasks by conditioning on a small set of input-output demonstrations.<n>We analyze how ICL performance is governed by (i) the quality of selected demonstrations, (ii) an intrinsic ICL capability of the pretrained model, and (iii) the degree of distribution shift.
• Score: 9.83462430479849
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In-Context Learning (ICL) enables pretrained LLMs to adapt to downstream tasks by conditioning on a small set of input-output demonstrations, without any parameter updates. Although there have been many theoretical efforts to explain how ICL works, most either rely on strong architectural or data assumptions, or fail to capture the impact of key practical factors such as demonstration selection, Chain-of-Thought (CoT) prompting, the number of demonstrations, and prompt templates. We address this gap by establishing a theoretical analysis of ICL under mild assumptions that links these design choices to generalization behavior. We derive an upper bound on the ICL test loss, showing that performance is governed by (i) the quality of selected demonstrations, quantified by Lipschitz constants of the ICL loss along paths connecting test prompts to pretraining samples, (ii) an intrinsic ICL capability of the pretrained model, and (iii) the degree of distribution shift. Within the same framework, we analyze CoT prompting as inducing a task decomposition and show that it is beneficial when demonstrations are well chosen at each substep and the resulting subtasks are easier to learn. Finally, we characterize how ICL performance sensitivity to prompt templates varies with the number of demonstrations. Together, our study shows that pretraining equips the model with the ability to generalize beyond observed tasks, while CoT enables the model to compose simpler subtasks into more complex ones, and demonstrations and instructions enable it to retrieve similar or complex tasks, including those that can be composed into more complex ones, jointly supporting generalization to unseen tasks. All theoretical insights are corroborated by experiments.

Related papers

• TwinICL: Diagnosing Multimodal In-Context Learning through Paired Counterfactuals [49.800443965874756]
  In-context learning (ICL) enables models to infer tasks from demonstrations.<n>Existing benchmarks generally lack matched text and image versions needed to compare ICL performance across modalities.<n>We introduce TwinICL, a procedurally generated benchmark providing such pairs for controlled comparison.
  arXiv  Detail & Related papers  (2026-09-14T04:42:42Z)
• Many-Shot CoT-ICL: Making In-Context Learning Truly Learn [58.439517684779936]
  In-context learning (ICL) adapts large language models to new tasks by conditioning on demonstrations in the prompt without parameter updates.<n>We study many-shot chain-of-thought in-context learning (CoT-ICL) for reasoning and show that standard many-shot rules do not transfer.
  arXiv  Detail & Related papers  (2026-05-13T13:30:12Z)
• Beyond the Prompt in Large Language Models: Comprehension, In-Context Learning, and Chain-of-Thought [15.598263332303612]
  Large Language Models (LLMs) have demonstrated remarkable proficiency across diverse tasks.<n>This study dives into the foundations of these observations by addressing three critical questions.
  arXiv  Detail & Related papers  (2026-02-16T10:57:26Z)
• From Compression to Expression: A Layerwise Analysis of In-Context Learning [24.45948310980883]
  In-context learning (ICL) enables large language models to adapt to new tasks without weight updates by learning from demonstration sequences.<n>We conduct a statistical geometric analysis of ICL representations to investigate how task-specific information is captured across layers.<n>Our findings reveal an intriguing layerwise dynamic in ICL, highlight how structured representations emerge within LLMs, and showcase that analyzing internal representations can facilitate a deeper understanding of model behavior.
  arXiv  Detail & Related papers  (2025-05-22T22:22:03Z)
• PICLe: Pseudo-Annotations for In-Context Learning in Low-Resource Named Entity Detection [56.916656013563355]
  In-context learning (ICL) enables Large Language Models to perform tasks using few demonstrations.<n>We propose PICLe, a framework for in-context learning with noisy, pseudo-annotated demonstrations.<n>We evaluate PICLe on five biomedical NED datasets and show that, with zero human annotation, PICLe outperforms ICL in low-resource settings.
  arXiv  Detail & Related papers  (2024-12-16T16:09:35Z)
• ParaICL: Towards Parallel In-Context Learning [74.38022919598443]
  Large language models (LLMs) have become the norm in natural language processing.<n>Few-shot in-context learning (ICL) relies on the choice of few-shot demonstration examples.<n>We propose a novel method named parallel in-context learning (ParaICL)
  arXiv  Detail & Related papers  (2024-03-31T05:56:15Z)
• Dr.ICL: Demonstration-Retrieved In-context Learning [29.142262267850704]
  In-context learning (ICL) teaching a large language model to perform a task with few-shot demonstrations has emerged as a strong paradigm for using LLMs. Recent research suggests that retrieving semantically similar demonstrations to the input from a pool of available demonstrations results in better performance. This work expands the applicability of retrieval-based ICL approaches by demonstrating that even simple word-overlap similarity measures such as BM25 outperform randomly selected demonstrations.
  arXiv  Detail & Related papers  (2023-05-23T14:55:25Z)
• Iterative Forward Tuning Boosts In-Context Learning in Language Models [88.25013390669845]
  In this study, we introduce a novel two-stage framework to boost in-context learning in large language models (LLMs) Specifically, our framework delineates the ICL process into two distinct stages: Deep-Thinking and test stages. The Deep-Thinking stage incorporates a unique attention mechanism, i.e., iterative enhanced attention, which enables multiple rounds of information accumulation.
  arXiv  Detail & Related papers  (2023-05-22T13:18:17Z)
• Enhancing Chain-of-Thoughts Prompting with Iterative Bootstrapping in Large Language Models [81.01397924280612]
  Large language models (LLMs) can achieve highly effective performance on various reasoning tasks by incorporating step-by-step chain-of-thought (CoT) prompting as demonstrations. We introduce Iter-CoT (Iterative bootstrapping in Chain-of-Thoughts Prompting), an iterative bootstrapping approach for selecting exemplars and generating reasoning chains.
  arXiv  Detail & Related papers  (2023-04-23T13:54:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.