PromptHub: Enhancing Multi-Prompt Visual In-Context Learning with Locality-Aware Fusion, Concentration and Alignment

• URL: http://arxiv.org/abs/2603.18891v1
• Date: Thu, 19 Mar 2026 13:30:17 GMT
• Title: PromptHub: Enhancing Multi-Prompt Visual In-Context Learning with Locality-Aware Fusion, Concentration and Alignment
• Abstract summary: Visual In-Context Learning (VICL) aims to complete vision tasks by imitating pixel demonstrations.<n>Recent work pioneered prompt fusion that combines the advantages of various demonstrations, which shows a promising way to extend VICL.<n>We introduce PromptHub, a framework that holistically strengthens multi-prompting through locality-aware fusion, concentration and alignment.
• Score: 87.39378178514092
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Visual In-Context Learning (VICL) aims to complete vision tasks by imitating pixel demonstrations. Recent work pioneered prompt fusion that combines the advantages of various demonstrations, which shows a promising way to extend VICL. Unfortunately, the patch-wise fusion framework and model-agnostic supervision hinder the exploitation of informative cues, thereby limiting performance gains. To overcome this deficiency, we introduce PromptHub, a framework that holistically strengthens multi-prompting through locality-aware fusion, concentration and alignment. PromptHub exploits spatial priors to capture richer contextual information, employs complementary concentration, alignment, and prediction objectives to mutually guide training, and incorporates data augmentation to further reinforce supervision. Extensive experiments on three fundamental vision tasks demonstrate the superiority of PromptHub. Moreover, we validate its universality, transferability, and robustness across out-of-distribution settings, and various retrieval scenarios. This work establishes a reliable locality-aware paradigm for prompt fusion, moving beyond prior patch-wise approaches. Code is available at https://github.com/luotc-why/ICLR26-PromptHub.

Related papers

• EVL-MCoT: Enhanced Vision-Language Multi-CoT for Harmful Meme Detection [13.261187830239797]
  MEMEs are widely used on the internet and often carry strong elements of sarcasm or irony.<n>Existing methods focus on the dual-stream vision-language model to extract the visual and text simultaneously.<n>We propose an enhanced vision-language multi-CoT (EVL-MCoT) approach to address these limitations.
  arXiv  Detail & Related papers  (2026-07-24T06:28:57Z)
• Latent Visual States for Efficient Multimodal Reasoning [58.350643620457]
  EVA (LatEnt Visual StAtes) is a novel framework that generates continuous latent visual representations.<n>These internal representations manifest as an adaptive sequence of Latent_slot tokens.<n> EVA achieves significant performance gains while enhancing inference efficiency.
  arXiv  Detail & Related papers  (2026-06-23T07:22:22Z)
• Learning Modal-Mixed Chain-of-Thought Reasoning with Latent Embeddings [39.4633015395276]
  We study how to extend chain-of-thought (CoT) beyond language to better handle multimodal reasoning.<n>We introduce modal-mixed CoT, which interleaves textual tokens with compact visual sketches represented as latent embeddings.<n>Our method yields better performance than language-only and other CoT methods.
  arXiv  Detail & Related papers  (2026-01-31T07:36:38Z)
• Enhancing Visual In-Context Learning by Multi-Faceted Fusion [6.852150407828682]
  We introduce a novel framework that moves beyond single-prompt fusion towards an multi-combination collaborative fusion.<n>Our method generates three contextual representation branches, each formed by integrating information from different combinations of top-quality prompts.<n>Experiments on diverse tasks, including foreground segmentation, single-object detection, and image colorization, highlight its strong cross-task generalization.
  arXiv  Detail & Related papers  (2026-01-15T06:25:09Z)
• True Multimodal In-Context Learning Needs Attention to the Visual Context [69.63677595066012]
  Multimodal Large Language Models (MLLMs) have enabled Multimodal In-Context Learning (MICL)-adapting to new tasks.<n>Current MLLMs tend to neglect visual cues and over-rely on textual patterns, leading to mere text imitation rather than genuine multimodal adaptation.<n>We introduce Dynamic Attention Reallocation (DARA), an efficient fine-tuning strategy that encourages models to attend to the visual context.
  arXiv  Detail & Related papers  (2025-07-21T17:08:18Z)
• ConViTac: Aligning Visual-Tactile Fusion with Contrastive Representations [7.870120920732663]
  We propose ConViTac, a visual-tactile representation learning network designed to enhance the alignment of features during fusion.<n>Our key contribution is a Contrastive Embedding Conditioning mechanism that leverages a contrastive encoder pretrained to project visual and tactile inputs into unified latent embeddings.<n>We conduct extensive experiments to demonstrate the superiority of ConViTac in real world over current state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-06-25T18:43:35Z)
• Every SAM Drop Counts: Embracing Semantic Priors for Multi-Modality Image Fusion and Beyond [52.486290612938895]
  We propose a novel method that leverages the semantic knowledge from the Segment Anything Model (SAM) to Grow the quality of fusion results and Enable downstream task adaptability.<n> Specifically, we design a Semantic Persistent Attention (SPA) Module that efficiently maintains source information via the persistent repository while extracting high-level semantic priors from SAM.<n>Our method achieves a balance between high-quality visual results and downstream task adaptability while maintaining practical deployment efficiency.
  arXiv  Detail & Related papers  (2025-03-03T06:16:31Z)
• ClawMachine: Learning to Fetch Visual Tokens for Referential Comprehension [71.03445074045092]
  We propose ClawMachine, offering a new methodology that explicitly notates each entity using token collectives groups of visual tokens.<n>Our method unifies the prompt and answer of visual referential tasks without using additional syntax.<n>ClawMachine achieves superior performance on scene-level and referential understanding tasks with higher efficiency.
  arXiv  Detail & Related papers  (2024-06-17T08:39:16Z)
• Attend and Enrich: Enhanced Visual Prompt for Zero-Shot Learning [114.59476118365266]
  We propose AENet, which endows semantic information into the visual prompt to distill semantic-enhanced prompt for visual representation enrichment.<n> AENet comprises two key steps: 1) exploring the concept-harmonized tokens for the visual and attribute modalities, grounded on the modal-sharing token that represents consistent visual-semantic concepts; and 2) yielding semantic-enhanced prompt via the visual residual refinement unit with attribute consistency supervision.
  arXiv  Detail & Related papers  (2024-06-05T07:59:48Z)
• Optimization of Prompt Learning via Multi-Knowledge Representation for Vision-Language Models [26.964848679914354]
  CoKnow is a framework that enhances Prompt Learning for Vision-Language Models with rich contextual knowledge.<n>We conducted extensive experiments on 11 publicly available datasets, demonstrating that CoKnow outperforms a series of previous methods.
  arXiv  Detail & Related papers  (2024-04-16T07:44:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.