Zero-Shot Sketch Based Image Retrieval using Graph Transformer

• URL: http://arxiv.org/abs/2201.10185v1
• Date: Tue, 25 Jan 2022 09:02:39 GMT
• Title: Zero-Shot Sketch Based Image Retrieval using Graph Transformer
• Authors: Sumrit Gupta, Ushasi Chaudhuri, Biplab Banerjee
• Abstract summary: We propose a novel graph transformer based zero-shot sketch-based image retrieval (GTZSR) framework for solving ZS-SBIR tasks. To bridge the domain gap between the visual features, we propose minimizing the Wasserstein distance between images and sketches in a learned domain-shared space. We also propose a novel compatibility loss that further aligns the two visual domains by bridging the domain gap of one class with respect to the domain gap of all other classes in the training set.
• Score: 18.00165431469872
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The performance of a zero-shot sketch-based image retrieval (ZS-SBIR) task is primarily affected by two challenges. The substantial domain gap between image and sketch features needs to be bridged, while at the same time the side information has to be chosen tactfully. Existing literature has shown that varying the semantic side information greatly affects the performance of ZS-SBIR. To this end, we propose a novel graph transformer based zero-shot sketch-based image retrieval (GTZSR) framework for solving ZS-SBIR tasks which uses a novel graph transformer to preserve the topology of the classes in the semantic space and propagates the context-graph of the classes within the embedding features of the visual space. To bridge the domain gap between the visual features, we propose minimizing the Wasserstein distance between images and sketches in a learned domain-shared space. We also propose a novel compatibility loss that further aligns the two visual domains by bridging the domain gap of one class with respect to the domain gap of all other classes in the training set. Experimental results obtained on the extended Sketchy, TU-Berlin, and QuickDraw datasets exhibit sharp improvements over the existing state-of-the-art methods in both ZS-SBIR and generalized ZS-SBIR.

Related papers

• Symmetrical Bidirectional Knowledge Alignment for Zero-Shot Sketch-Based Image Retrieval [69.46139774646308]
  This paper studies the problem of zero-shot sketch-based image retrieval (ZS-SBIR) It aims to use sketches from unseen categories as queries to match the images of the same category. We propose a novel Symmetrical Bidirectional Knowledge Alignment for zero-shot sketch-based image retrieval (SBKA)
  arXiv  Detail & Related papers  (2023-12-16T04:50:34Z)
• Adapt and Align to Improve Zero-Shot Sketch-Based Image Retrieval [85.39613457282107]
  Cross-domain nature of sketch-based image retrieval is challenging. We present an effective Adapt and Align'' approach to address the key challenges. Inspired by recent advances in image-text foundation models (e.g., CLIP) on zero-shot scenarios, we explicitly align the learned image embedding with a more semantic text embedding to achieve the desired knowledge transfer from seen to unseen classes.
  arXiv  Detail & Related papers  (2023-05-09T03:10:15Z)
• BDA-SketRet: Bi-Level Domain Adaptation for Zero-Shot SBIR [52.78253400327191]
  BDA-SketRet is a novel framework performing a bi-level domain adaptation for aligning the spatial and semantic features of the visual data pairs. Experimental results on the extended Sketchy, TU-Berlin, and QuickDraw exhibit sharp improvements over the literature.
  arXiv  Detail & Related papers  (2022-01-17T18:45:55Z)
• Domain-Smoothing Network for Zero-Shot Sketch-Based Image Retrieval [66.37346493506737]
  Zero-Shot Sketch-Based Image Retrieval (ZS-SBIR) is a novel cross-modal retrieval task. We propose a novel Domain-Smoothing Network (DSN) for ZS-SBIR. Our approach notably outperforms the state-of-the-art methods in both Sketchy and TU-Berlin datasets.
  arXiv  Detail & Related papers  (2021-06-22T14:58:08Z)
• Towards Unsupervised Sketch-based Image Retrieval [126.77787336692802]
  We introduce a novel framework that simultaneously performs unsupervised representation learning and sketch-photo domain alignment. Our framework achieves excellent performance in the new unsupervised setting, and performs comparably or better than state-of-the-art in the zero-shot setting.
  arXiv  Detail & Related papers  (2021-05-18T02:38:22Z)
• CrossATNet - A Novel Cross-Attention Based Framework for Sketch-Based Image Retrieval [30.249581102239645]
  We propose a novel framework for cross-modal zero-shot learning (ZSL) in the context of sketch-based image retrieval (SBIR) While we define a cross-modal triplet loss to ensure the discriminative nature of the shared space, an innovative cross-modal attention learning strategy is also proposed to guide feature extraction from the image domain.
  arXiv  Detail & Related papers  (2021-04-20T12:11:12Z)
• Semantically Tied Paired Cycle Consistency for Any-Shot Sketch-based Image Retrieval [55.29233996427243]
  Low-shot sketch-based image retrieval is an emerging task in computer vision. In this paper, we address any-shot, i.e. zero-shot and few-shot, sketch-based image retrieval (SBIR) tasks. For solving these tasks, we propose a semantically aligned cycle-consistent generative adversarial network (SEM-PCYC) Our results demonstrate a significant boost in any-shot performance over the state-of-the-art on the extended version of the Sketchy, TU-Berlin and QuickDraw datasets.
  arXiv  Detail & Related papers  (2020-06-20T22:43:53Z)
• Progressive Domain-Independent Feature Decomposition Network for Zero-Shot Sketch-Based Image Retrieval [15.955284712628444]
  We propose a Progressive Domain-independent Feature Decomposition (PDFD) network for ZS-SBIR. Specifically, PDFD decomposes visual features into domain features and semantic ones, and then the semantic features are projected into common space as retrieval features for ZS-SBIR.
  arXiv  Detail & Related papers  (2020-03-22T12:07:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.