UCDR-Adapter: Exploring Adaptation of Pre-Trained Vision-Language Models for Universal Cross-Domain Retrieval

• URL: http://arxiv.org/abs/2412.10680v1
• Date: Sat, 14 Dec 2024 04:59:38 GMT
• Title: UCDR-Adapter: Exploring Adaptation of Pre-Trained Vision-Language Models for Universal Cross-Domain Retrieval
• Authors: Haoyu Jiang, Zhi-Qi Cheng, Gabriel Moreira, Jiawen Zhu, Jingdong Sun, Bukun Ren, Jun-Yan He, Qi Dai, Xian-Sheng Hua,
• Abstract summary: Universal Cross-Domain Retrieval (UCDR) retrieves relevant images from unseen domains and classes without semantic labels. We propose UCDR-Adapter, which enhances pre-trained models with adapters and dynamic prompt generation.
• Score: 36.64936080387957
• License:
• Abstract: Universal Cross-Domain Retrieval (UCDR) retrieves relevant images from unseen domains and classes without semantic labels, ensuring robust generalization. Existing methods commonly employ prompt tuning with pre-trained vision-language models but are inherently limited by static prompts, reducing adaptability. We propose UCDR-Adapter, which enhances pre-trained models with adapters and dynamic prompt generation through a two-phase training strategy. First, Source Adapter Learning integrates class semantics with domain-specific visual knowledge using a Learnable Textual Semantic Template and optimizes Class and Domain Prompts via momentum updates and dual loss functions for robust alignment. Second, Target Prompt Generation creates dynamic prompts by attending to masked source prompts, enabling seamless adaptation to unseen domains and classes. Unlike prior approaches, UCDR-Adapter dynamically adapts to evolving data distributions, enhancing both flexibility and generalization. During inference, only the image branch and generated prompts are used, eliminating reliance on textual inputs for highly efficient retrieval. Extensive benchmark experiments show that UCDR-Adapter consistently outperforms ProS in most cases and other state-of-the-art methods on UCDR, U(c)CDR, and U(d)CDR settings.

Related papers

• CILP-FGDI: Exploiting Vision-Language Model for Generalizable Person Re-Identification [42.429118831928214]
  We explore the use of CLIP (Contrastive Language-Image Pretraining), a vision-language model pretrained on large-scale image-text pairs to align visual and textual features. The adaptation of CLIP to the task presents two primary challenges: learning more fine-grained features to enhance discriminative ability, and learning more domain-invariant features to improve the model's generalization capabilities.
  arXiv  Detail & Related papers  (2025-01-27T14:08:25Z)
• CFPL-FAS: Class Free Prompt Learning for Generalizable Face Anti-spoofing [66.6712018832575]
  Domain generalization (DG) based Face Anti-Spoofing (FAS) aims to improve the model's performance on unseen domains. We make use of large-scale VLMs like CLIP and leverage the textual feature to dynamically adjust the classifier's weights for exploring generalizable visual features.
  arXiv  Detail & Related papers  (2024-03-21T11:58:50Z)
• ProS: Prompting-to-simulate Generalized knowledge for Universal Cross-Domain Retrieval [123.51277978744677]
  We propose textbfPrompting-to-textbfSimulate (ProS) to apply prompt tuning for Universal Cross-Domain Retrieval (UCDR) ProS employs a two-step process to simulate Content-aware Dynamic Prompts (CaDP) which can impact models to produce generalized features for UCDR. Our method achieves new state-of-the-art performance without bringing excessive parameters.
  arXiv  Detail & Related papers  (2023-12-19T14:39:11Z)
• Learning Domain Invariant Prompt for Vision-Language Models [31.581652862478965]
  We propose a novel prompt learning paradigm that directly generates emphdomain invariant prompt that can be generalized to unseen domains, called MetaPrompt. Our method consistently and significantly outperforms existing methods.
  arXiv  Detail & Related papers  (2022-12-08T11:23:24Z)
• P{\O}DA: Prompt-driven Zero-shot Domain Adaptation [27.524962843495366]
  We adapt a model trained on a source domain using only a general description in natural language of the target domain, i.e., a prompt. We show that these prompt-driven augmentations can be used to perform zero-shot domain adaptation for semantic segmentation.
  arXiv  Detail & Related papers  (2022-12-06T18:59:58Z)
• DSE-GAN: Dynamic Semantic Evolution Generative Adversarial Network for Text-to-Image Generation [71.87682778102236]
  We propose a novel Dynamical Semantic Evolution GAN (DSE-GAN) to re-compose each stage's text features under a novel single adversarial multi-stage architecture. DSE-GAN achieves 7.48% and 37.8% relative FID improvement on two widely used benchmarks.
  arXiv  Detail & Related papers  (2022-09-03T06:13:26Z)
• Refign: Align and Refine for Adaptation of Semantic Segmentation to Adverse Conditions [78.71745819446176]
  Refign is a generic extension to self-training-based UDA methods which leverages cross-domain correspondences. Refign consists of two steps: (1) aligning the normal-condition image to the corresponding adverse-condition image using an uncertainty-aware dense matching network, and (2) refining the adverse prediction with the normal prediction using an adaptive label correction mechanism. The approach introduces no extra training parameters, minimal computational overhead -- during training only -- and can be used as a drop-in extension to improve any given self-training-based UDA method.
  arXiv  Detail & Related papers  (2022-07-14T11:30:38Z)
• CLIP-Adapter: Better Vision-Language Models with Feature Adapters [79.52844563138493]
  We show that there is an alternative path to achieve better vision-language models other than prompt tuning. In this paper, we propose CLIP-Adapter to conduct fine-tuning with feature adapters on either visual or language branch. Experiments and extensive ablation studies on various visual classification tasks demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2021-10-09T11:39:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.