MAG: MAnifold Guided Semi-Supervised Multi-modal In-Context Learning

• URL: http://arxiv.org/abs/2608.12724v1
• Date: Thu, 13 Aug 2026 02:10:52 GMT
• Title: MAG: MAnifold Guided Semi-Supervised Multi-modal In-Context Learning
• Abstract summary: In-context learning with multi-modal large language models (MLLMs) enables task adaptation without parameter updates.<n>Unlabeled multi-modal data is abundant, but it remains elusive how to exploit them for ICL.<n>We propose MAG (MAnifold-Guided semi-supervised in-context demonstra- tion selection), an efficient framework that leverages unlabeled data to improve ICL.
• Score: 48.467551522475276
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Few-shot in-context learning (ICL) with multi-modal large language models (MLLMs) enables task adaptation without parameter updates, but its performance is highly sensitive to the quality and coverage of the selected demonstrations. While unlabeled multi-modal data is abundant, it remains elusive how to exploit them for ICL. We propose MAG (MAnifold-Guided semi-supervised in-context demonstra- tion selection), an efficient framework that leverages unlabeled data to improve multi-modal ICL. MAG formulates demonstration selection as a semi-supervised propagation problem on a multi-modal graph and adopts a two-stage strategy: (i) relevance score propagation identifies a compact set of high-impact unlabeled samples for pseudo-labeling, reducing MLLM inference cost; (ii) multi-modal relevance is used to select the final demonstrations. We show that textual represen- tations are more effective for relevance propagation, while both visual and textual modalities are crucial for high-quality demonstration selection. Experiments on eight multi-modal benchmarks demonstrate that MAG consistently outperforms strong baselines in label-scarce regimes, achieving significant gains with a limited pseudo-labeling budget.

Related papers

• MMLDSum-LLM: Multimodal Long-Document Summarization with Visual-Alignment and Keyword-Aware [11.787892687988368]
  We introduce MMLDSum-Bench, a high-quality benchmark for multimodal long-document summarization.<n>We also propose MMLDSum-LLM, a reproducible two-stage training framework.<n>Our approach significantly improves key-information coverage and cross-modal consistency.
  arXiv  Detail & Related papers  (2026-07-30T10:54:09Z)
• MAF: Multimodal Adaptive Few-shot Prompting for Sentiment Analysis with MLLMs [0.0]
  We propose a Multimodal Adaptive Few-Shot Prompting (MAF) framework for sentiment analysis.<n>MAF retrieves and integrates query-relevant demonstrations to elicit the sentiment reasoning capabilities of MLLMs in a context-sensitive manner.<n>Experiments on public benchmark datasets demonstrate that MAF achieves substantial and consistent performance improvements.
  arXiv  Detail & Related papers  (2026-06-14T09:16:36Z)
• Beyond Unimodal Shortcuts: MLLMs as Cross-Modal Reasoners for Grounded Named Entity Recognition [51.68340973140949]
  Multimodal Named Entity Recognition (GMNER) aims to extract text-based entities, assign them semantic categories, and ground them to corresponding visual regions.<n> MLLMs exhibit $textbfmodality bias$, including visual bias and textual bias, which stems from their tendency to take unimodal shortcuts.<n>We propose Modality-aware Consistency Reasoning ($bfMCR$), which enforces structured cross-modal reasoning.
  arXiv  Detail & Related papers  (2026-02-04T12:12:49Z)
• Learn to Select: Exploring Label Distribution Divergence for In-Context Demonstration Selection in Text Classification [9.105555204653275]
  In-context learning (ICL) for text classification has demonstrated impressive performance on large language models (LLMs)<n>We propose a two-stage demonstration selection method, TopK + Label Distribution Divergence (L2D)<n>This enables the selection of demonstrations that are not only semantically similar but also aligned in label distribution with the test input.
  arXiv  Detail & Related papers  (2025-11-10T08:04:14Z)
• MAPLE: Many-Shot Adaptive Pseudo-Labeling for In-Context Learning [53.02571749383208]
  In-Context Learning (ICL) empowers Large Language Models (LLMs) to tackle diverse tasks by incorporating multiple input-output examples.<n>Many-Shot Adaptive Pseudo-LabEling (MAPLE) is a novel influence-based many-shot ICL framework that utilizes pseudo-labeled samples to compensate for the lack of label information.
  arXiv  Detail & Related papers  (2025-05-22T04:54:27Z)
• NoteLLM-2: Multimodal Large Representation Models for Recommendation [71.87790090964734]
  Large Language Models (LLMs) have demonstrated exceptional proficiency in text understanding and embedding tasks.<n>Their potential in multimodal representation, particularly for item-to-item (I2I) recommendations, remains underexplored.<n>We propose an end-to-end fine-tuning method that customizes the integration of any existing LLMs and vision encoders for efficient multimodal representation.
  arXiv  Detail & Related papers  (2024-05-27T03:24:01Z)
• A Unified Optimal Transport Framework for Cross-Modal Retrieval with Noisy Labels [22.2715520667186]
  Cross-modal retrieval (CMR) aims to establish interaction between different modalities. This work proposes UOT-RCL, a Unified framework based on Optimal Transport (OT) for Robust Cross-modal Retrieval. Experiments on three widely-used cross-modal retrieval datasets demonstrate that our UOT-RCL surpasses the state-of-the-art approaches.
  arXiv  Detail & Related papers  (2024-03-20T10:34:40Z)
• MMICT: Boosting Multi-Modal Fine-Tuning with In-Context Examples [63.78384552789171]
  This paper introduces Multi-Modal In-Context Tuning (MMICT), a novel multi-modal fine-tuning paradigm. We propose the Multi-Modal Hub (M-Hub), a unified module that captures various multi-modal features according to different inputs and objectives. Based on M-Hub, MMICT enables MM-LLMs to learn from in-context visual-guided textual features and subsequently generate outputs conditioned on the textual-guided visual features.
  arXiv  Detail & Related papers  (2023-12-11T13:11:04Z)
• Retrieval-augmented Multi-modal Chain-of-Thoughts Reasoning for Large Language Models [56.256069117502385]
  Chain of Thought (CoT) approaches can be used to enhance the capability of Large Language Models (LLMs) on complex reasoning tasks. However, the selection of optimal CoT demonstration examples in multi-modal reasoning remains less explored. We introduce a novel approach that addresses this challenge by using retrieval mechanisms to automatically select demonstration examples.
  arXiv  Detail & Related papers  (2023-12-04T08:07:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.