Hyper-ICL: Attention Calibration with Hyperbolic Anchor Distillation for Multimodal In-Context Learning

• URL: http://arxiv.org/abs/2606.04434v1
• Date: Wed, 03 Jun 2026 04:32:09 GMT
• Title: Hyper-ICL: Attention Calibration with Hyperbolic Anchor Distillation for Multimodal In-Context Learning
• Abstract summary: Hyper-ICL is a training-based framework for demonstration-free multimodal ICL.<n>It reconstructs demonstration effects directly without requiring ICDs at inference time.<n>It consistently improves accuracy and stability over vanilla ICL and existing state-of-the-art methods.
• Score: 10.293023172874213
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal In-Context Learning (ICL) has emerged as a practical inference paradigm for Multimodal Large Language Models, where a small set of interleaved image-text In-Context Demonstrations (ICDs) conditions the model to solve new tasks. Despite its flexibility, multimodal ICL incurs high inference latency and suffers from instability due to sensitivity to demonstration formatting, ordering, and content. To address these limitations, we propose Hyper-ICL, a lightweight, training-based framework for demonstration-free multimodal ICL that reconstructs demonstration effects directly without requiring ICDs at inference time. Hyper-ICL learns a parameter-efficient low-rank logit-level adapter that calibrates attention distributions to better match demonstration-induced attention redistribution. To capture how demonstration influence varies across queries, we introduce a query-adaptive modulation mechanism that adaptively controls intervention strength at token level across layers and heads based on the current query. Finally, we propose a layer-wise hyperbolic anchor distillation loss that aligns intermediate student features to a demonstration-conditioned teacher via Lorentz geodesic distance. This loss encourages the student to reconstruct the demonstration-query relationships induced by ICDs. Extensive experiments across six different multimodal benchmarks (including VQAv2, OK-VQA, and COCO Caption) demonstrate that Hyper-ICL consistently improves accuracy and stability over vanilla ICL and existing state-of-the-art methods.

Related papers

• Enhancing Multimodal In-Context Learning via Inductive-Deductive Reasoning [65.15766304205657]
  In-context learning (ICL) allows large models to adapt to tasks using a few examples, yet its extension to vision-language models (VLMs) remains fragile.<n>Our analysis reveals that the fundamental limitation lies in an inductive gap, models often produce correct answers from flawed reasoning.<n>We introduce a framework that restructures multimodal ICL as a principled inductive-deductive process.
  arXiv  Detail & Related papers  (2026-05-04T09:18:19Z)
• UniICL: Systematizing Unified Multimodal In-context Learning through a Capability-Oriented Taxonomy [99.95663439922937]
  In-context Learning efficacy is often non-monotonic and highly task-dependent.<n>We introduce a six-level capability-oriented taxonomy that categorizes the functional role of demonstrations.<n>We construct UniICL-760K, a large-scale corpus featuring curated 8-shot In-context Learning episodes.<n>As an architectural intervention, we propose the Context-Adaptive Prototype Modulator.
  arXiv  Detail & Related papers  (2026-03-25T18:09:33Z)
• Steering Vision-Language-Action Models as Anti-Exploration: A Test-Time Scaling Approach [78.4812458793128]
  We propose textbfTACO, a test-time-scaling framework that applies a lightweight pseudo-count estimator as a high-fidelity verifier of action chunks.<n>Our method resembles the classical anti-exploration principle in offline reinforcement learning (RL), and being gradient-free, it incurs significant computational benefits.
  arXiv  Detail & Related papers  (2025-12-02T14:42:54Z)
• Drift No More? Context Equilibria in Multi-Turn LLM Interactions [58.69551510148673]
  contexts drift is the gradual divergence of a model's outputs from goal-consistent behavior across turns.<n>Unlike single-turn errors, drift unfolds temporally and is poorly captured by static evaluation metrics.<n>We show that multi-turn drift can be understood as a controllable equilibrium phenomenon rather than as inevitable decay.
  arXiv  Detail & Related papers  (2025-10-09T04:48:49Z)
• Multimodal LLM-Guided Semantic Correction in Text-to-Image Diffusion [52.315729095824906]
  MLLM Semantic-Corrected Ping-Pong-Ahead Diffusion (PPAD) is a novel framework that introduces a Multimodal Large Language Model (MLLM) as a semantic observer during inference.<n>It performs real-time analysis on intermediate generations, identifies latent semantic inconsistencies, and translates feedback into controllable signals that actively guide the remaining denoising steps.<n>Extensive experiments demonstrate PPAD's significant improvements.
  arXiv  Detail & Related papers  (2025-05-26T14:42:35Z)
• CAMA: Enhancing Multimodal In-Context Learning with Context-Aware Modulated Attention [32.07189678228538]
  Multimodal in-context learning (ICL) is emerging as a key capability that enables large vision-language models (LVLMs) to adapt to novel tasks without parameter updates.<n>ICL remains unstable, even with well-matched in-context demonstrations (ICDs), suggesting that LVLMs struggle to fully utilize the provided context.<n>We propose textbfContext-Aware Modulated Attention (CAMA), a plug-and-play and training-free method that dynamically modulates LVLM's attention logits based on the input in-context sequence.
  arXiv  Detail & Related papers  (2025-05-21T04:25:23Z)
• Advancing Multimodal In-Context Learning in Large Vision-Language Models with Task-aware Demonstrations [0.0]
  Multimodal in-context learning (ICL) has emerged as a key capability of Large Vision-Language Models (LVLMs)<n>We shed light on the core mechanism underlying multimodal ICL, identifying task mapping as a crucial factor in configuring robust in-context demonstration sequences.<n>We propose textitSabER, a lightweight yet powerful decoder-only transformer equipped with task-aware attention.
  arXiv  Detail & Related papers  (2025-03-05T16:33:10Z)
• Focused Large Language Models are Stable Many-Shot Learners [18.783939647966776]
  In-Context Learning (ICL) enables large language models (LLMs) to achieve rapid task adaptation by learning from demonstrations. We propose a training-free method FocusICL, which conducts triviality filtering to avoid attention being diverted by unimportant contents. We show that FocusICL achieves an average performance improvement of 5.2% over vanilla ICL and scales well with many-shot demonstrations.
  arXiv  Detail & Related papers  (2024-08-26T02:53:24Z)
• SSLCL: An Efficient Model-Agnostic Supervised Contrastive Learning Framework for Emotion Recognition in Conversations [20.856739541819056]
  Emotion recognition in conversations (ERC) is a rapidly evolving task within the natural language processing community. We propose an efficient and model-agnostic SCL framework named Supervised Sample-Label Contrastive Learning with Soft-HGR Maximal Correlation (SSLCL) We introduce a novel perspective on utilizing label representations by projecting discrete labels into dense embeddings through a shallow multilayer perceptron.
  arXiv  Detail & Related papers  (2023-10-25T14:41:14Z)
• Scaling In-Context Demonstrations with Structured Attention [75.41845145597875]
  We propose a better architectural design for in-context learning. Structured Attention for In-Context Learning replaces the full-attention by a structured attention mechanism. We show that SAICL achieves comparable or better performance than full attention while obtaining up to 3.4x inference speed-up.
  arXiv  Detail & Related papers  (2023-07-05T23:26:01Z)
• Iterative Forward Tuning Boosts In-Context Learning in Language Models [88.25013390669845]
  In this study, we introduce a novel two-stage framework to boost in-context learning in large language models (LLMs) Specifically, our framework delineates the ICL process into two distinct stages: Deep-Thinking and test stages. The Deep-Thinking stage incorporates a unique attention mechanism, i.e., iterative enhanced attention, which enables multiple rounds of information accumulation.
  arXiv  Detail & Related papers  (2023-05-22T13:18:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.