When Is a Task Vector Enough? An Empirical Theory of Implicit Multimodal ICL

• URL: http://arxiv.org/abs/2608.13385v1
• Date: Thu, 13 Aug 2026 15:46:50 GMT
• Title: When Is a Task Vector Enough? An Empirical Theory of Implicit Multimodal ICL
• Abstract summary: Implicit multimodal in-context learning compresses demonstrations into internal interventions, ranging from static task vectors to query-conditioned transformations and attention routing.<n>We propose the Selection--Realization Hypothesis, which views demonstrations as inducing a compact family of internal changes from which the query selects.<n>By contrasting correct demonstrations with matched counterfactuals, we measure the structure of explicit M-ICL and test whether it predicts intervention behavior.
• Score: 2.964628733433076
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Implicit multimodal in-context learning compresses demonstrations into internal interventions, ranging from static task vectors to query-conditioned transformations and attention routing. Despite their common goal, these methods differ substantially in how the intervention depends on the query and where it modifies the model, leaving unclear which additional complexity is necessary for a given task. We propose the Selection--Realization Hypothesis. It views demonstrations as inducing a compact family of internal changes from which the query selects, while the model's computation constrains how the selected change can be implemented. We evaluate this account using controlled multimodal tasks in which query dependence varies without changing the underlying task primitives or prompt format. By contrasting correct demonstrations with matched counterfactuals, we measure the structure of explicit M-ICL and test whether it predicts intervention behavior. We find that the success of a static task vector is closely tied to how much of the demonstration-induced change is shared across queries. Additional intervention complexity becomes useful when explicit M-ICL contains query-specific or distributed structure that a local additive shift cannot recover. These relationships extend to natural VQA benchmarks and support cost-aware method selection without access to test performance. Our results provide a unified empirical theory of when demonstrations can be compressed into a task vector and when a more expressive intervention is warranted.

Related papers

• TwinICL: Diagnosing Multimodal In-Context Learning through Paired Counterfactuals [49.800443965874756]
  In-context learning (ICL) enables models to infer tasks from demonstrations.<n>Existing benchmarks generally lack matched text and image versions needed to compare ICL performance across modalities.<n>We introduce TwinICL, a procedurally generated benchmark providing such pairs for controlled comparison.
  arXiv  Detail & Related papers  (2026-09-14T04:42:42Z)
• When Correct Demonstrations Hurt: Rethinking the Role of Exemplars in In-Context Learning [14.102553871638243]
  In-context learning (ICL) is often motivated by the intuition that demonstrations help because they provide correct input-output examples.<n>We show that correctness does not guarantee exemplar utility, and some correct demonstrations can even reduce ICL accuracy.<n>Our results show that robust ICL requires evaluating not only whether demonstrations are correct, but also how they influence contextual inference.
  arXiv  Detail & Related papers  (2026-05-25T21:52:49Z)
• Many-Shot CoT-ICL: Making In-Context Learning Truly Learn [58.439517684779936]
  In-context learning (ICL) adapts large language models to new tasks by conditioning on demonstrations in the prompt without parameter updates.<n>We study many-shot chain-of-thought in-context learning (CoT-ICL) for reasoning and show that standard many-shot rules do not transfer.
  arXiv  Detail & Related papers  (2026-05-13T13:30:12Z)
• Demonstrations, CoT, and Prompting: A Theoretical Analysis of ICL [9.83462430479849]
  In-Context Learning (ICL) enables pretrained LLMs to adapt to downstream tasks by conditioning on a small set of input-output demonstrations.<n>We analyze how ICL performance is governed by (i) the quality of selected demonstrations, (ii) an intrinsic ICL capability of the pretrained model, and (iii) the degree of distribution shift.
  arXiv  Detail & Related papers  (2026-03-20T03:33:04Z)
• Theoretical Foundations of Prompt Engineering: From Heuristics to Expressivity [0.0]
  We study the family of functions obtainable by holding a Transformer backbone fixed as an executor and varying only the prompt.<n>We prove a constructive existential result showing that a single fixed backbone can approximate a broad class of target behaviors via prompts alone.
  arXiv  Detail & Related papers  (2025-12-14T13:42:20Z)
• Where and What Matters: Sensitivity-Aware Task Vectors for Many-Shot Multimodal In-Context Learning [57.082554323521464]
  We propose a Sensitivity-aware Task Vector insertion framework (STV) to figure out where and what to insert.<n>Our key insight is that activation deltas across query-context pairs exhibit consistent structural patterns, providing a reliable cue for insertion.<n>Based on the identified sensitive-aware locations, we construct a pre-clustered activation bank for each location by clustering the activation values, and then apply reinforcement learning to choose the most suitable one to insert.
  arXiv  Detail & Related papers  (2025-11-11T13:42:13Z)
• Adaptive Task Vectors for Large Language Models [14.108866468832623]
  Adaptive Task Vectors (ATV) is a simple and effective framework that dynamically generates task vectors conditioned on each input query.<n>ATV demonstrates strong performance and generalization capabilities, even for unseen tasks.
  arXiv  Detail & Related papers  (2025-06-03T22:12:28Z)
• Does learning the right latent variables necessarily improve in-context learning? [13.168329639763678]
  Large autoregressive models like Transformers can solve tasks through in-context learning (ICL) without learning new weights.<n>In this paper, we investigate the effect of explicitly inferring task latents.<n>We find little discernible difference between the two; biasing towards task-relevant latent variables does not lead to better out-of-distribution performance.
  arXiv  Detail & Related papers  (2024-05-29T15:06:10Z)
• Prompt Optimization with EASE? Efficient Ordering-aware Automated Selection of Exemplars [66.823588073584]
  Large language models (LLMs) have shown impressive capabilities in real-world applications. The quality of these exemplars in the prompt greatly impacts performance. Existing methods fail to adequately account for the impact of exemplar ordering on the performance.
  arXiv  Detail & Related papers  (2024-05-25T08:23:05Z)
• ParaICL: Towards Parallel In-Context Learning [74.38022919598443]
  Large language models (LLMs) have become the norm in natural language processing.<n>Few-shot in-context learning (ICL) relies on the choice of few-shot demonstration examples.<n>We propose a novel method named parallel in-context learning (ParaICL)
  arXiv  Detail & Related papers  (2024-03-31T05:56:15Z)
• Adaptive Discrete Communication Bottlenecks with Dynamic Vector Quantization [76.68866368409216]
  We propose learning to dynamically select discretization tightness conditioned on inputs. We show that dynamically varying tightness in communication bottlenecks can improve model performance on visual reasoning and reinforcement learning tasks.
  arXiv  Detail & Related papers  (2022-02-02T23:54:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.