CMAP: Cross-Modal Adaptive Prompting for Multi-Domain Task-Incremental Learning

• URL: http://arxiv.org/abs/2605.25708v1
• Date: Mon, 25 May 2026 11:09:48 GMT
• Title: CMAP: Cross-Modal Adaptive Prompting for Multi-Domain Task-Incremental Learning
• Abstract summary: Text-space task routing replaces visual Gaussian matching with cosine similarity to frozen CLIP text prototypes.<n>Visual-textual confidence replaces single-Gaussian class modeling with K-means visual prototypes and cross-modal alignment scores under task-calibrated thresholds.<n>On the MTIL benchmark spanning 118.7% and 1201 classes, our method achieves 74.2% Transfer, 80.5% Average, and 8 Last under Order-I.
• Score: 3.1368611610608856
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multi-domain task-incremental learning requires a model to sequentially acquire knowledge across visually diverse domains without forgetting prior tasks, and without access to task identity at inference. Parameter-efficient methods built on frozen vision-language models have made strong progress, yet all existing approaches rely exclusively on visual features for task routing, confidence estimation, and encoder adaptation, leaving CLIP's cross-modal text embedding space entirely unexploited. We address this gap through three contributions. Text-space task routing replaces visual Gaussian matching with cosine similarity to frozen CLIP text prototypes, giving order-independent routing robust to data scarcity at zero parameter cost. Multi-prototype visual-textual confidence replaces single-Gaussian class modeling with K-means visual prototypes and cross-modal alignment scores under task-calibrated thresholds. Symmetric cross-modal gating extends per-layer Gumbel gates to the text encoder conditioned on batch image features, preserving cross-modal alignment on out-of-distribution inputs. On the MTIL benchmark spanning 11 datasets and 1201 classes, our method achieves 74.2% Transfer, 80.5% Average, and 88.7% Last under Order-I, surpassing the prior state of the art by 5.0, 3.7, and 3.0 percentage points with only 2.5M trainable parameters and no external data.

Related papers

• FLAT: Resampling Image and Text into 1D Flexible-Length Aligned Transmodal Tokens for Retrieval and Generation [29.59094476146176]
  Flat is a representation pre-training framework that jointly optimize a shared multimodal encoder alongside downstream text-to-image (T2I) and image-to-text (I2T) decoders.<n>A single pre-training stage allows FLAT to perform cross-modal retrieval and generation across variable prefix K, achieving a T2I GenEval score of 71.1.
  arXiv  Detail & Related papers  (2026-09-15T03:35:59Z)
• Visual Information Extraction from Documents via Classification-Guided Large Vision-Language Models [7.176656402971074]
  We propose a classification-guided large vision-language model (LVLM) framework for multi-type VIE that achieves high accuracy with minimal supervision.<n>The framework offers an efficient, scalable solution for complex document understanding in office automation.
  arXiv  Detail & Related papers  (2026-07-22T03:27:50Z)
• COVTrack++: Learning Open-Vocabulary Multi-Object Tracking from Continuous Videos via a Synergistic Paradigm [59.26203051651017]
  C-TAO is the first continuously annotated training set for Open-Vocabulary Multi-Object Tracking (OVMOT)<n>For the framework bottleneck, we propose COVTrack++, a synergistic framework that achieves a bidirectional reciprocal mechanism between detection and association through three modules.<n>Experiments on TAO demonstrate state-of-the-art performance, with novel TETA reaching 35.4% and 30.5% on validation and test sets, improving novel AssocA by 4.8% and novel LocA by 5.8% over previous methods.
  arXiv  Detail & Related papers  (2026-03-25T07:20:27Z)
• MMLoP: Multi-Modal Low-Rank Prompting for Efficient Vision-Language Adaptation [12.481603155570037]
  We propose textbfMMLoP (textbfMulti-textbfModal textbfLow-Rank textbfPrompting), a framework that achieves deep multi-modal prompting with only textbf11.5K trainable parameters.
  arXiv  Detail & Related papers  (2026-02-24T22:00:34Z)
• MM-ACT: Learn from Multimodal Parallel Generation to Act [80.9182259389658]
  MM-ACT integrates text, image, and action in shared token space and performs generation across all three modalities.<n> Context-Shared Multimodal Learning supervises generation in all three modalities from a shared context.<n>Our approach achieves a success rate of 96.3% on LIBERO, 72.0% across three tasks of real Franka, and 52.38% across eight bimanual tasks of RoboTwin2.0.
  arXiv  Detail & Related papers  (2025-11-30T16:46:35Z)
• Learning Hyperspectral Images with Curated Text Prompts for Efficient Multimodal Alignment [1.7188280334580195]
  We make an attempt to optimize a Vision-Language Model (VLM) for hyperspectral scene understanding by exploiting a CLIP-style contrastive training framework.<n>Our framework maps voxel-level embeddings from a vision backbone onto the latent space of a frozen large embedding model.<n>It is seen that the proposed method updates only 0.07 percent of the total parameters, yet yields state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-09-20T23:23:04Z)
• Mario at EXIST 2025: A Simple Gateway to Effective Multilingual Sexism Detection [8.40042895828361]
  EXIST 2025 Task 1 addresses text-based sexism detection in English and Spanish tweets through hierarchical Low-Rank Adaptation (LoRA) of Llama 3.1 8B.<n>Our method introduces conditional adapter routing that explicitly models dependencies across three hierarchically structured subtasks.<n>Our approach reduces training time by 75% and model storage by 98%, while achieving competitive performance across all subtasks.
  arXiv  Detail & Related papers  (2025-07-15T05:30:32Z)
• Leveraging Cross-Modal Neighbor Representation for Improved CLIP Classification [54.96876797812238]
  We present a novel CrOss-moDal nEighbor Representation(CODER) based on the distance structure between images and their neighbor texts. The key to construct a high-quality CODER lies in how to create a vast amount of high-quality and diverse texts to match with images. Experiment results across various datasets and models confirm CODER's effectiveness.
  arXiv  Detail & Related papers  (2024-04-27T02:04:36Z)
• Hierarchical Side-Tuning for Vision Transformers [33.536948382414316]
  Fine-tuning pre-trained Vision Transformers (ViTs) has showcased significant promise in enhancing visual recognition tasks. PETL has shown potential for achieving high performance with fewer parameter updates compared to full fine-tuning. This paper introduces Hierarchical Side-Tuning (HST), an innovative PETL method facilitating the transfer of ViT models to diverse downstream tasks.
  arXiv  Detail & Related papers  (2023-10-09T04:16:35Z)
• Cross-Modal Multi-Tasking for Speech-to-Text Translation via Hard Parameter Sharing [72.56219471145232]
  We propose a ST/MT multi-tasking framework with hard parameter sharing. Our method reduces the speech-text modality gap via a pre-processing stage. We show that our framework improves attentional encoder-decoder, Connectionist Temporal Classification (CTC), transducer, and joint CTC/attention models by an average of +0.5 BLEU.
  arXiv  Detail & Related papers  (2023-09-27T17:48:14Z)
• Prompt Tuning for Parameter-efficient Medical Image Segmentation [79.09285179181225]
  We propose and investigate several contributions to achieve a parameter-efficient but effective adaptation for semantic segmentation on two medical imaging datasets. We pre-train this architecture with a dedicated dense self-supervision scheme based on assignments to online generated prototypes. We demonstrate that the resulting neural network model is able to attenuate the gap between fully fine-tuned and parameter-efficiently adapted models.
  arXiv  Detail & Related papers  (2022-11-16T21:55:05Z)
• Meta-Generating Deep Attentive Metric for Few-shot Classification [53.07108067253006]
  We present a novel deep metric meta-generation method to generate a specific metric for a new few-shot learning task. In this study, we structure the metric using a three-layer deep attentive network that is flexible enough to produce a discriminative metric for each task. We gain surprisingly obvious performance improvement over state-of-the-art competitors, especially in the challenging cases.
  arXiv  Detail & Related papers  (2020-12-03T02:07:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.