How Few-Shot Examples Add Up: A Causal Decomposition of Function Vectors in In-Context Learning

• URL: http://arxiv.org/abs/2605.16591v1
• Date: Fri, 15 May 2026 19:49:56 GMT
• Title: How Few-Shot Examples Add Up: A Causal Decomposition of Function Vectors in In-Context Learning
• Abstract summary: In-context learning (ICL) excels at new tasks from minimal examples.<n>We show how few-shot prompts shape a model's function vector (FV)<n>We show that models contextualize individual examples' representations based on prior examples to adaptively reweight which demonstrations dominate the FV.
• Score: 48.80008450455555
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In-context learning (ICL) excels at new tasks from minimal examples, yet we still lack a mechanistic explanation of how few-shot prompts shape a model's function vector (FV)--a causal activation direction that drives task behavior on the ICL query. Across tasks and models, an $n$-shot FV is well-approximated by a linear combination of example-level sub-FVs, suggesting additive and composable contributions from individual demonstrations. Beyond additivity, we show that models contextualize individual examples' representations based on prior examples to adaptively reweight which demonstrations dominate the FV: attention shifts toward examples that are more informative and less ambiguous under the context. Finally, a causal decomposition separates Query-Key routing from Value updates, finding that contextualization's most consistent contributions to FV quality arise from Query-Key alignment--particularly in ambiguous settings--while Value-mediated effects are more heterogeneous. Together, these results unify additive superposition with context-dependent attention reweighting into a mechanistic, testable account of how few-shot prompts implement tasks.

Related papers

• Enhancing Multimodal In-Context Learning via Inductive-Deductive Reasoning [65.15766304205657]
  In-context learning (ICL) allows large models to adapt to tasks using a few examples, yet its extension to vision-language models (VLMs) remains fragile.<n>Our analysis reveals that the fundamental limitation lies in an inductive gap, models often produce correct answers from flawed reasoning.<n>We introduce a framework that restructures multimodal ICL as a principled inductive-deductive process.
  arXiv  Detail & Related papers  (2026-05-04T09:18:19Z)
• Retrieving Counterfactuals Improves Visual In-Context Learning [41.6338086518055]
  In-context learning (ICL) offers a promising avenue for vision-language models to adapt to new tasks.<n>Existing retrieval-augmented approaches rely on passive similarity-based retrieval.<n>We introduce CIRCLES, a novel framework that actively constructs demonstration sets by retrieving counterfactual-style examples.
  arXiv  Detail & Related papers  (2026-03-17T16:18:09Z)
• Contextualize-then-Aggregate: Circuits for In-Context Learning in Gemma-2 2B [51.74607395697567]
  In-Context Learning (ICL) is an intriguing ability of large language models (LLMs)<n>We use causal interventions to identify information flow in Gemma-2 2B for five naturalistic ICL tasks.<n>We find that the model infers task information using a two-step strategy we call contextualize-then-aggregate.
  arXiv  Detail & Related papers  (2025-03-31T18:33:55Z)
• Sub-SA: Strengthen In-context Learning via Submodular Selective Annotation [4.846839863393725]
  We propose Sub-SA (Submodular Selective ), a sub-module-based selective annotation method. The aim of Sub-SA is to reduce annotation costs while improving the quality of in-context examples. We also propose RPR (Reward and Penalty Regularization) to better balance the diversity and representativeness of the unlabeled dataset.
  arXiv  Detail & Related papers  (2024-07-08T07:47:30Z)
• DoubleDipper: Improving Long-Context LLMs via Context Recycling [44.24067814871803]
  We propose DoubleDipper, a novel In-Context-Learning method that automatically generates few-shot examples for long context QA tasks.<n>We apply our method on multiple Large Language Models and obtain substantial improvements.<n>Surprisingly, despite introducing only single-hop ICL examples, LLMs successfully generalize to multi-hop long-context QA.
  arXiv  Detail & Related papers  (2024-06-19T15:28:29Z)
• Prompt Optimization with EASE? Efficient Ordering-aware Automated Selection of Exemplars [66.823588073584]
  Large language models (LLMs) have shown impressive capabilities in real-world applications. The quality of these exemplars in the prompt greatly impacts performance. Existing methods fail to adequately account for the impact of exemplar ordering on the performance.
  arXiv  Detail & Related papers  (2024-05-25T08:23:05Z)
• Addressing Order Sensitivity of In-Context Demonstration Examples in Causal Language Models [18.03259038587496]
  In-context learning can be significantly influenced by the order of in-context demonstration examples. We introduce an unsupervised fine-tuning method, termed the Information-Augmented and Consistency-Enhanced approach. Our proposed method can reduce the sensitivity of CausalLMs to the order of in-context examples and exhibit robust generalizability.
  arXiv  Detail & Related papers  (2024-02-23T22:39:12Z)
• In-Context Probing: Toward Building Robust Classifiers via Probing Large Language Models [5.5089506884366735]
  In this paper, we propose an alternative approach, which we term In-Context Probing (ICP) Similar to in-context learning, we contextualize the representation of the input with an instruction, but instead of decoding the output prediction, we probe the contextualized representation to predict the label. We show that ICP performs competitive or superior to finetuning and can be particularly helpful to build classifiers on top of smaller models.
  arXiv  Detail & Related papers  (2023-05-23T15:43:04Z)
• K-Shot Contrastive Learning of Visual Features with Multiple Instance Augmentations [67.46036826589467]
  $K$-Shot Contrastive Learning is proposed to investigate sample variations within individual instances. It aims to combine the advantages of inter-instance discrimination by learning discriminative features to distinguish between different instances. Experiment results demonstrate the proposed $K$-shot contrastive learning achieves superior performances to the state-of-the-art unsupervised methods.
  arXiv  Detail & Related papers  (2020-07-27T04:56:41Z)
• Learning What Makes a Difference from Counterfactual Examples and Gradient Supervision [57.14468881854616]
  We propose an auxiliary training objective that improves the generalization capabilities of neural networks. We use pairs of minimally-different examples with different labels, a.k.a counterfactual or contrasting examples, which provide a signal indicative of the underlying causal structure of the task. Models trained with this technique demonstrate improved performance on out-of-distribution test sets.
  arXiv  Detail & Related papers  (2020-04-20T02:47:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.