SeCo-SBIR: Semantically Consistent Prompt Learning for Zero-Shot Sketch-Based Image Retrieval

• URL: http://arxiv.org/abs/2608.03120v1
• Date: Tue, 04 Aug 2026 04:41:45 GMT
• Title: SeCo-SBIR: Semantically Consistent Prompt Learning for Zero-Shot Sketch-Based Image Retrieval
• Abstract summary: SeCo-SBIR is a semantically consistent prompt learning framework for zero-shot sketch-based image retrieval.<n>It achieves state-of-the-art results on all three standard ZS-SBIR benchmarks across categorical, generalized, and across-dataset settings.
• Score: 2.018258593552284
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Adapting CLIP for zero-shot sketch-based image retrieval (ZS-SBIR) via prompt learning faces a fundamental tension: the model must bridge the sketch-photo domain gap through task-specific adaptation, yet the added flexibility risks overfitting to seen training categories and eroding CLIP's zero-shot generalization. We present SeCo-SBIR, a semantically consistent prompt learning framework that resolves this tension from both sides. First, a text-guided multi-modal prompting strategy routes learnable prompt vectors through CLIP's text encoder and projects the resulting intermediate representations into the visual encoder at every layer via learnable coupling functions. Because the text encoder has already learned robust, abstract category-level semantics from large-scale language supervision, this mechanism injects transferable semantic knowledge directly into the visual pathway - adapting the model to the sketch-photo domain while inherently favoring generalization to unseen classes. Second, a perturbation-based consistency constraint addresses the residual overfitting risk from the learnable coupling functions by aligning the adapted model with a frozen CLIP reference branch using an asymmetric InfoNCE objective - augmented inputs feed the frozen branch while clean inputs feed the trainable branch - anchoring the learned representations to CLIP's generalizable feature space. Together with lightweight adapters and a multi-objective loss combining triplet, NT-Xent, and classification terms, SeCo-SBIR achieves state-of-the-art results on all three standard ZS-SBIR benchmarks across categorical, generalized, and across-dataset settings.

Related papers

• DiCE-CIR: Direct Composition Learning for Efficient Zero-Shot Composed Image Retrieval [52.15696141238419]
  Zero-shot composed image retrieval (ZS-CIR) aims to retrieve a target image from a multimodal query consisting of a reference image and an edit text.<n>We propose DiCE-CIR, a direct composition learning method that predicts composed query representations by directly composing a reference image and an edit text.
  arXiv  Detail & Related papers  (2026-07-06T04:37:30Z)
• Decoupling Endpoint and Semantic Transition Learning for Zero-Shot Composed Image Retrieval [61.75093083322011]
  We show that DeCIR consistently improves projection-based ZS-CIR without increasing inference complexity.<n>Experiments on CIRR, CIRCO, FashionIQ, and GeneCIS demonstrate that DeCIR consistently improves projection-based ZS-CIR without increasing inference complexity.
  arXiv  Detail & Related papers  (2026-05-08T18:55:41Z)
• CLNet: Cross-View Correspondence Makes a Stronger Geo-Localizationer [48.52152634356309]
  We propose a correspondence-aware feature refinement framework, termed CLNet, that explicitly bridges the semantic and geometric gaps between different views.<n> CLNet decomposes the view alignment process into three learnable and complementary modules.<n>Our proposed CLNet achieves state-of-the-art performance while offering better interpretability and generalizability.
  arXiv  Detail & Related papers  (2025-12-16T16:31:41Z)
• SETR: A Two-Stage Semantic-Enhanced Framework for Zero-Shot Composed Image Retrieval [4.230223288110963]
  Zero-shot Composed Image Retrieval (ZS-CIR) aims to retrieve a target image given a reference image and a relative text, without relying on costly triplet annotations.<n>Existing CLIP-based methods face two core challenges: (1) union-based feature fusion indiscriminately aggregates all visual cues, carrying over irrelevant background details that dilute the intended modification, and (2) global cosine similarity from CLIP embeddings lacks the ability to resolve fine-grained semantic relations.
  arXiv  Detail & Related papers  (2025-09-30T09:41:52Z)
• DART: Dual Adaptive Refinement Transfer for Open-Vocabulary Multi-Label Recognition [59.203152078315235]
  Open-Vocabulary Multi-Label Recognition (OV-MLR) aims to identify multiple seen and unseen object categories within an image.<n> Vision-Language Pre-training models offer a strong open-vocabulary foundation, but struggle with fine-grained localization under weak supervision.<n>We propose the Dual Adaptive Refinement Transfer (DART) framework to overcome these limitations.
  arXiv  Detail & Related papers  (2025-08-07T17:22:33Z)
• DiSa: Directional Saliency-Aware Prompt Learning for Generalizable Vision-Language Models [5.027492394254859]
  DiSa is a Directional Saliency-Aware Prompt Learning framework.<n>It integrates two complementary regularization strategies to enhance generalization.<n>It consistently outperforms state-of-the-art prompt learning methods across various settings.
  arXiv  Detail & Related papers  (2025-05-26T00:14:52Z)
• SketchYourSeg: Mask-Free Subjective Image Segmentation via Freehand Sketches [116.1810651297801]
  SketchYourSeg establishes freehand sketches as a powerful query modality for subjective image segmentation.<n>Our evaluations demonstrate superior performance over existing approaches across diverse benchmarks.
  arXiv  Detail & Related papers  (2025-01-27T13:07:51Z)
• Elevating All Zero-Shot Sketch-Based Image Retrieval Through Multimodal Prompt Learning [11.033050922826934]
  We introduce SpLIP, a novel multi-modal prompt learning scheme designed to operate with frozen CLIP backbones. SpLIP implements a bi-directional prompt-sharing strategy that enables mutual knowledge exchange between CLIP's visual and textual encoders. We propose two innovative strategies for further refining the embedding space.
  arXiv  Detail & Related papers  (2024-07-05T01:30:42Z)
• No Token Left Behind: Explainability-Aided Image Classification and Generation [79.4957965474334]
  We present a novel explainability-based approach, which adds a loss term to ensure that CLIP focuses on all relevant semantic parts of the input. Our method yields an improvement in the recognition rate, without additional training or fine-tuning.
  arXiv  Detail & Related papers  (2022-04-11T07:16:39Z)
• Semantically Tied Paired Cycle Consistency for Any-Shot Sketch-based Image Retrieval [55.29233996427243]
  Low-shot sketch-based image retrieval is an emerging task in computer vision. In this paper, we address any-shot, i.e. zero-shot and few-shot, sketch-based image retrieval (SBIR) tasks. For solving these tasks, we propose a semantically aligned cycle-consistent generative adversarial network (SEM-PCYC) Our results demonstrate a significant boost in any-shot performance over the state-of-the-art on the extended version of the Sketchy, TU-Berlin and QuickDraw datasets.
  arXiv  Detail & Related papers  (2020-06-20T22:43:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.