CLAIR: CLIP-Aided Weakly Supervised Zero-Shot Cross-Domain Image Retrieval

• URL: http://arxiv.org/abs/2508.12290v1
• Date: Sun, 17 Aug 2025 08:43:45 GMT
• Title: CLAIR: CLIP-Aided Weakly Supervised Zero-Shot Cross-Domain Image Retrieval
• Authors: Chor Boon Tan, Conghui Hu, Gim Hee Lee,
• Abstract summary: We propose CLAIR to refine the noisy pseudo-labels with a confidence score from the similarity between the CLIP text and image features.<n>We also learn a novel cross-domain mapping function in closed-form, using only CLIP text embeddings.<n>Our CLAIR consistently shows superior performance compared to existing state-of-the-art methods.
• Score: 51.45839347847945
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The recent growth of large foundation models that can easily generate pseudo-labels for huge quantity of unlabeled data makes unsupervised Zero-Shot Cross-Domain Image Retrieval (UZS-CDIR) less relevant. In this paper, we therefore turn our attention to weakly supervised ZS-CDIR (WSZS-CDIR) with noisy pseudo labels generated by large foundation models such as CLIP. To this end, we propose CLAIR to refine the noisy pseudo-labels with a confidence score from the similarity between the CLIP text and image features. Furthermore, we design inter-instance and inter-cluster contrastive losses to encode images into a class-aware latent space, and an inter-domain contrastive loss to alleviate domain discrepancies. We also learn a novel cross-domain mapping function in closed-form, using only CLIP text embeddings to project image features from one domain to another, thereby further aligning the image features for retrieval. Finally, we enhance the zero-shot generalization ability of our CLAIR to handle novel categories by introducing an extra set of learnable prompts. Extensive experiments are carried out using TUBerlin, Sketchy, Quickdraw, and DomainNet zero-shot datasets, where our CLAIR consistently shows superior performance compared to existing state-of-the-art methods.

Related papers

• Spherical Linear Interpolation and Text-Anchoring for Zero-shot Composed Image Retrieval [43.47770490199544]
  Composed Image Retrieval (CIR) is a complex task that retrieves images using a query, which is configured with an image and a caption. We introduce a novel ZS-CIR method that uses Spherical Linear Interpolation (Slerp) to directly merge image and text representations. We also introduce Text-Anchored-Tuning (TAT), a method that fine-tunes the image encoder while keeping the text encoder fixed.
  arXiv  Detail & Related papers  (2024-05-01T15:19:54Z)
• CLIP-Guided Source-Free Object Detection in Aerial Images [17.26407623526735]
  High-resolution aerial images often require substantial storage space and may not be readily accessible to the public. We propose a novel Source-Free Object Detection (SFOD) method to address these challenges. To alleviate the noisy labels in self-training, we utilize Contrastive Language-Image Pre-training (CLIP) to guide the generation of pseudo-labels. By leveraging CLIP's zero-shot classification capability, we aggregate its scores with the original predicted bounding boxes, enabling us to obtain refined scores for the pseudo-labels.
  arXiv  Detail & Related papers  (2024-01-10T14:03:05Z)
• ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehension [114.85628613911713]
  Large-scale pre-trained models are useful for image classification across domains. We present ReCLIP, a simple but strong zero-shot baseline that repurposes CLIP, a state-of-the-art large-scale model, for ReC.
  arXiv  Detail & Related papers  (2022-04-12T17:55:38Z)
• No Token Left Behind: Explainability-Aided Image Classification and Generation [79.4957965474334]
  We present a novel explainability-based approach, which adds a loss term to ensure that CLIP focuses on all relevant semantic parts of the input. Our method yields an improvement in the recognition rate, without additional training or fine-tuning.
  arXiv  Detail & Related papers  (2022-04-11T07:16:39Z)
• Zero-Shot Sketch Based Image Retrieval using Graph Transformer [18.00165431469872]
  We propose a novel graph transformer based zero-shot sketch-based image retrieval (GTZSR) framework for solving ZS-SBIR tasks. To bridge the domain gap between the visual features, we propose minimizing the Wasserstein distance between images and sketches in a learned domain-shared space. We also propose a novel compatibility loss that further aligns the two visual domains by bridging the domain gap of one class with respect to the domain gap of all other classes in the training set.
  arXiv  Detail & Related papers  (2022-01-25T09:02:39Z)
• BDA-SketRet: Bi-Level Domain Adaptation for Zero-Shot SBIR [52.78253400327191]
  BDA-SketRet is a novel framework performing a bi-level domain adaptation for aligning the spatial and semantic features of the visual data pairs. Experimental results on the extended Sketchy, TU-Berlin, and QuickDraw exhibit sharp improvements over the literature.
  arXiv  Detail & Related papers  (2022-01-17T18:45:55Z)
• ACNet: Approaching-and-Centralizing Network for Zero-Shot Sketch-Based Image Retrieval [28.022137537238425]
  We propose an textbfApproaching-and-textbfCentralizing textbfNetwork (termed textbfACNet'') to jointly optimize sketch-to-photo synthesis and the image retrieval. The retrieval module guides the synthesis module to generate large amounts of diverse photo-like images which gradually approach the photo domain. Our approach achieves state-of-the-art performance on two widely used ZS-SBIR datasets and surpasses previous methods by a large margin.
  arXiv  Detail & Related papers  (2021-11-24T19:36:10Z)
• Domain-Smoothing Network for Zero-Shot Sketch-Based Image Retrieval [66.37346493506737]
  Zero-Shot Sketch-Based Image Retrieval (ZS-SBIR) is a novel cross-modal retrieval task. We propose a novel Domain-Smoothing Network (DSN) for ZS-SBIR. Our approach notably outperforms the state-of-the-art methods in both Sketchy and TU-Berlin datasets.
  arXiv  Detail & Related papers  (2021-06-22T14:58:08Z)
• Towards Unsupervised Sketch-based Image Retrieval [126.77787336692802]
  We introduce a novel framework that simultaneously performs unsupervised representation learning and sketch-photo domain alignment. Our framework achieves excellent performance in the new unsupervised setting, and performs comparably or better than state-of-the-art in the zero-shot setting.
  arXiv  Detail & Related papers  (2021-05-18T02:38:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.