Enhancing Visual In-Context Learning by Multi-Faceted Fusion

• URL: http://arxiv.org/abs/2601.10107v1
• Date: Thu, 15 Jan 2026 06:25:09 GMT
• Title: Enhancing Visual In-Context Learning by Multi-Faceted Fusion
• Authors: Wenwen Liao, Jianbo Yu, Yuansong Wang, Qingchao Jiang, Xiaofeng Yang,
• Abstract summary: We introduce a novel framework that moves beyond single-prompt fusion towards an multi-combination collaborative fusion.<n>Our method generates three contextual representation branches, each formed by integrating information from different combinations of top-quality prompts.<n>Experiments on diverse tasks, including foreground segmentation, single-object detection, and image colorization, highlight its strong cross-task generalization.
• Score: 6.852150407828682
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Visual In-Context Learning (VICL) has emerged as a powerful paradigm, enabling models to perform novel visual tasks by learning from in-context examples. The dominant "retrieve-then-prompt" approach typically relies on selecting the single best visual prompt, a practice that often discards valuable contextual information from other suitable candidates. While recent work has explored fusing the top-K prompts into a single, enhanced representation, this still simply collapses multiple rich signals into one, limiting the model's reasoning capability. We argue that a more multi-faceted, collaborative fusion is required to unlock the full potential of these diverse contexts. To address this limitation, we introduce a novel framework that moves beyond single-prompt fusion towards an multi-combination collaborative fusion. Instead of collapsing multiple prompts into one, our method generates three contextual representation branches, each formed by integrating information from different combinations of top-quality prompts. These complementary guidance signals are then fed into proposed MULTI-VQGAN architecture, which is designed to jointly interpret and utilize collaborative information from multiple sources. Extensive experiments on diverse tasks, including foreground segmentation, single-object detection, and image colorization, highlight its strong cross-task generalization, effective contextual fusion, and ability to produce more robust and accurate predictions than existing methods.

Related papers

• Beyond Single Prompts: Synergistic Fusion and Arrangement for VICL [4.215181054941225]
  Vision In-Context Learning (VICL) enables inpainting models to quickly adapt to new visual tasks from only a few prompts.<n>VICL methods suffer from two key issues: (1) selecting only the most similar prompt discards complementary cues from other high-quality prompts; and (2) failing to exploit the structured information implied by different prompt arrangements.<n>We propose an end-to-end VICL framework to overcome these limitations. Firstly, an adaptive Fusion Module aggregates critical patterns and annotations from multiple prompts to form more precise contextual prompts.
  arXiv  Detail & Related papers  (2026-01-15T06:53:59Z)
• Query-Kontext: An Unified Multimodal Model for Image Generation and Editing [53.765351127477224]
  Unified Multimodal Models (UMMs) have demonstrated remarkable performance in text-to-image generation (T2I) and editing (TI2I)<n>We introduce Query-Kontext, a novel approach that bridges the VLM and diffusion model via a multimodal kontext'' composed of semantic cues and coarse-grained image conditions encoded from multimodal inputs.<n> Experiments show that our approach matches strong unified baselines and even outperforms task-specific state-of-the-art methods in several cases.
  arXiv  Detail & Related papers  (2025-09-30T17:59:46Z)
• UniMoCo: Unified Modality Completion for Robust Multi-Modal Embeddings [9.344107676552408]
  We propose UniMoCo, a vision-language model architecture designed for multi-modal embedding tasks.<n>We develop a specialized training strategy to align embeddings from both original and modality-completed inputs.<n>Experiments show that UniMoCo outperforms previous methods while demonstrating consistent robustness across diverse settings.
  arXiv  Detail & Related papers  (2025-05-17T03:53:11Z)
• X-Reflect: Cross-Reflection Prompting for Multimodal Recommendation [46.76427517818944]
  Cross-Reflection Prompting is designed to explicitly identify and reconcile supportive and conflicting information between text and images.<n>Experiments conducted on two widely used benchmarks demonstrate that our method outperforms existing prompting baselines in downstream recommendation accuracy.<n>We also introduce X-Reflect-keyword, a lightweight variant that summarizes image content using keywords and replaces the base model with a smaller backbone.
  arXiv  Detail & Related papers  (2024-08-27T16:10:21Z)
• NoteLLM-2: Multimodal Large Representation Models for Recommendation [71.87790090964734]
  Large Language Models (LLMs) have demonstrated exceptional proficiency in text understanding and embedding tasks.<n>Their potential in multimodal representation, particularly for item-to-item (I2I) recommendations, remains underexplored.<n>We propose an end-to-end fine-tuning method that customizes the integration of any existing LLMs and vision encoders for efficient multimodal representation.
  arXiv  Detail & Related papers  (2024-05-27T03:24:01Z)
• Unified Multi-modal Unsupervised Representation Learning for Skeleton-based Action Understanding [62.70450216120704]
  Unsupervised pre-training has shown great success in skeleton-based action understanding. We propose a Unified Multimodal Unsupervised Representation Learning framework, called UmURL. UmURL exploits an efficient early-fusion strategy to jointly encode the multi-modal features in a single-stream manner.
  arXiv  Detail & Related papers  (2023-11-06T13:56:57Z)
• Exploiting Modality-Specific Features For Multi-Modal Manipulation Detection And Grounding [54.49214267905562]
  We construct a transformer-based framework for multi-modal manipulation detection and grounding tasks. Our framework simultaneously explores modality-specific features while preserving the capability for multi-modal alignment. We propose an implicit manipulation query (IMQ) that adaptively aggregates global contextual cues within each modality.
  arXiv  Detail & Related papers  (2023-09-22T06:55:41Z)
• UniDoc: A Universal Large Multimodal Model for Simultaneous Text Detection, Recognition, Spotting and Understanding [93.92313947913831]
  We introduce UniDoc, a novel multimodal model equipped with text detection and recognition capabilities. To the best of our knowledge, this is the first large multimodal model capable of simultaneous text detection, recognition, spotting, and understanding.
  arXiv  Detail & Related papers  (2023-08-19T17:32:34Z)
• Multi-Grained Multimodal Interaction Network for Entity Linking [65.30260033700338]
  Multimodal entity linking task aims at resolving ambiguous mentions to a multimodal knowledge graph. We propose a novel Multi-GraIned Multimodal InteraCtion Network $textbf(MIMIC)$ framework for solving the MEL task.
  arXiv  Detail & Related papers  (2023-07-19T02:11:19Z)
• Multi-Prompt with Depth Partitioned Cross-Modal Learning [25.239388488952375]
  Partitioned Multi-modal Prompt (PMPO) is a multi-modal prompting technique that extends the soft prompt from a single learnable prompt to multiple prompts. Our method divides the visual encoder depths and connects learnable prompts to the separated visual depths, enabling different prompts to capture hierarchical contextual depths. We evaluate the effectiveness of our approach on three challenging tasks: new class generalization, cross-dataset evaluation, and domain generalization.
  arXiv  Detail & Related papers  (2023-05-10T14:54:29Z)
• High-Modality Multimodal Transformer: Quantifying Modality & Interaction Heterogeneity for High-Modality Representation Learning [112.51498431119616]
  This paper studies efficient representation learning for high-modality scenarios involving a large set of diverse modalities. A single model, HighMMT, scales up to 10 modalities (text, image, audio, video, sensors, proprioception, speech, time-series, sets, and tables) and 15 tasks from 5 research areas.
  arXiv  Detail & Related papers  (2022-03-02T18:56:20Z)
• Latent Structures Mining with Contrastive Modality Fusion for Multimedia Recommendation [22.701371886522494]
  We argue that the latent semantic item-item structures underlying multimodal contents could be beneficial for learning better item representations. We devise a novel modality-aware structure learning module, which learns item-item relationships for each modality.
  arXiv  Detail & Related papers  (2021-11-01T03:37:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.