TwinICL: Diagnosing Multimodal In-Context Learning through Paired Counterfactuals

• URL: http://arxiv.org/abs/2609.15028v1
• Date: Mon, 14 Sep 2026 04:42:42 GMT
• Title: TwinICL: Diagnosing Multimodal In-Context Learning through Paired Counterfactuals
• Abstract summary: In-context learning (ICL) enables models to infer tasks from demonstrations.<n>Existing benchmarks generally lack matched text and image versions needed to compare ICL performance across modalities.<n>We introduce TwinICL, a procedurally generated benchmark providing such pairs for controlled comparison.
• Score: 49.800443965874756
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In-context learning (ICL) enables models to infer tasks from demonstrations, but existing benchmarks generally lack matched text and image versions needed to compare ICL performance across modalities. We introduce TwinICL, a procedurally generated benchmark providing such pairs for controlled comparison. Across six open-weight models and 38 tasks, multimodal ICL consistently underperforms text-only ICL, with gaps varying by task family. To test whether this gap can be recovered, we target visual access, task framing, and reasoning through three interventions. Their combination recovers strong multimodal ICL performance on a diagnostic subset, despite limited or inconsistent individual effects. To distinguish difficulties in executing tasks from those in inferring them, we evaluate models with explicit task instructions, revealing a modality gap even when the task is known. We then examine how adding demonstration inputs and outputs reshapes this gap, highlighting demonstrations' dual role as additional context to process and evidence about the task. The dataset is available at https://github.com/lab-flair/TwinICL.

Related papers

• Many-Shot CoT-ICL: Making In-Context Learning Truly Learn [58.439517684779936]
  In-context learning (ICL) adapts large language models to new tasks by conditioning on demonstrations in the prompt without parameter updates.<n>We study many-shot chain-of-thought in-context learning (CoT-ICL) for reasoning and show that standard many-shot rules do not transfer.
  arXiv  Detail & Related papers  (2026-05-13T13:30:12Z)
• Why Multimodal In-Context Learning Lags Behind? Unveiling the Inner Mechanisms and Bottlenecks [7.62772056485722]
  In-context learning (ICL) enables models to adapt to new tasks via inference-time demonstrations.<n>Despite its success in large language models, the extension of ICL to multimodal settings remains poorly understood.<n>We show that multimodal ICL performs comparably to text-only ICL in zero-shot settings but degrades significantly under few-shot demonstrations.
  arXiv  Detail & Related papers  (2026-04-15T02:12:51Z)
• Stable Diffusion Models are Secretly Good at Visual In-Context Learning [9.829303881652548]
  We show that off-the-shelf Stable Diffusion models can be repurposed for visual in-context learning (V-ICL)<n>We formulate an in-place attention re-computation within the self-attention layers of the Stable Diffusion architecture.<n>We show that this repurposed Stable Diffusion model is able to adapt to six different tasks.
  arXiv  Detail & Related papers  (2025-08-13T17:08:22Z)
• Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMs [49.42020616826156]
  Vision-Language models (VLMs) show impressive abilities to answer questions on visual inputs, yet demonstrate higher accuracies when performing an analogous task on text.<n>We investigate this accuracy gap by identifying and comparing the textitcircuits in different modalities.<n>To overcome this, we patch the representations of visual data tokens from later layers back into earlier layers.
  arXiv  Detail & Related papers  (2025-06-10T17:59:21Z)
• Contextualize-then-Aggregate: Circuits for In-Context Learning in Gemma-2 2B [51.74607395697567]
  In-Context Learning (ICL) is an intriguing ability of large language models (LLMs)<n>We use causal interventions to identify information flow in Gemma-2 2B for five naturalistic ICL tasks.<n>We find that the model infers task information using a two-step strategy we call contextualize-then-aggregate.
  arXiv  Detail & Related papers  (2025-03-31T18:33:55Z)
• On Many-Shot In-Context Learning for Long-Context Evaluation [10.500629810624769]
  This paper delves into long-context language model evaluation through many-shot ICL.<n>We develop metrics to categorize ICL tasks into two groups: similar-sample learning (SSL) and all-sample learning (ASL)<n>We find that while state-of-the-art models demonstrate good performance up to 64k tokens in SSL tasks, many models experience significant performance drops at only 16k tokens in ASL tasks.
  arXiv  Detail & Related papers  (2024-11-11T17:00:59Z)
• X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoning [109.9413329636322]
  This paper introduces an efficient framework that integrates multiple modalities (images, 3D, audio and video) to a frozen Large Language Models (LLMs) Our approach explores two distinct projection mechanisms: Q-Formers and Linear Projections (LPs)
  arXiv  Detail & Related papers  (2023-11-30T18:43:51Z)
• ICL-D3IE: In-Context Learning with Diverse Demonstrations Updating for Document Information Extraction [56.790794611002106]
  Large language models (LLMs) have demonstrated remarkable results in various natural language processing (NLP) tasks with in-context learning. We propose a simple but effective in-context learning framework called ICL-D3IE. Specifically, we extract the most difficult and distinct segments from hard training documents as hard demonstrations.
  arXiv  Detail & Related papers  (2023-03-09T06:24:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.