Sketch3T: Test-Time Training for Zero-Shot SBIR

• URL: http://arxiv.org/abs/2203.14691v1
• Date: Mon, 28 Mar 2022 12:44:49 GMT
• Title: Sketch3T: Test-Time Training for Zero-Shot SBIR
• Authors: Aneeshan Sain, Ayan Kumar Bhunia, Vaishnav Potlapalli, Pinaki Nath Chowdhury, Tao Xiang, Yi-Zhe Song
• Abstract summary: Zero-shot sketch-based image retrieval typically asks for a trained model to be applied as is to unseen categories. We extend ZS-SBIR asking it to transfer to both categories and sketch distributions. Our key contribution is a test-time training paradigm that can adapt using just one sketch.
• Score: 106.59164595640704
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Zero-shot sketch-based image retrieval typically asks for a trained model to be applied as is to unseen categories. In this paper, we question to argue that this setup by definition is not compatible with the inherent abstract and subjective nature of sketches, i.e., the model might transfer well to new categories, but will not understand sketches existing in different test-time distribution as a result. We thus extend ZS-SBIR asking it to transfer to both categories and sketch distributions. Our key contribution is a test-time training paradigm that can adapt using just one sketch. Since there is no paired photo, we make use of a sketch raster-vector reconstruction module as a self-supervised auxiliary task. To maintain the fidelity of the trained cross-modal joint embedding during test-time update, we design a novel meta-learning based training paradigm to learn a separation between model updates incurred by this auxiliary task from those off the primary objective of discriminative learning. Extensive experiments show our model to outperform state of-the-arts, thanks to the proposed test-time adaption that not only transfers to new categories but also accommodates to new sketching styles.

Related papers

• SketchGPT: Autoregressive Modeling for Sketch Generation and Recognition [4.6519578789100215]
  SketchGPT is a flexible framework that employs a sequence-to-sequence autoregressive model for sketch generation, and completion. By mapping complex sketches into simplified sequences of abstract primitives, our approach significantly streamlines the input for autoregressive modeling.
  arXiv  Detail & Related papers  (2024-05-06T01:24:14Z)
• It's All About Your Sketch: Democratising Sketch Control in Diffusion Models [114.73766136068357]
  This paper unravels the potential of sketches for diffusion models, addressing the deceptive promise of direct sketch control in generative AI. We importantly democratise the process, enabling amateur sketches to generate precise images, living up to the commitment of "what you sketch is what you get"
  arXiv  Detail & Related papers  (2024-03-12T01:05:25Z)
• Symmetrical Bidirectional Knowledge Alignment for Zero-Shot Sketch-Based Image Retrieval [69.46139774646308]
  This paper studies the problem of zero-shot sketch-based image retrieval (ZS-SBIR) It aims to use sketches from unseen categories as queries to match the images of the same category. We propose a novel Symmetrical Bidirectional Knowledge Alignment for zero-shot sketch-based image retrieval (SBKA)
  arXiv  Detail & Related papers  (2023-12-16T04:50:34Z)
• Customize StyleGAN with One Hand Sketch [0.0]
  We propose a framework to control StyleGAN imagery with a single user sketch. We learn a conditional distribution in the latent space of a pre-trained StyleGAN model via energy-based learning. Our model can generate multi-modal images semantically aligned with the input sketch.
  arXiv  Detail & Related papers  (2023-10-29T09:32:33Z)
• Active Learning for Fine-Grained Sketch-Based Image Retrieval [1.994307489466967]
  The ability to retrieve a photo by mere free-hand sketching highlights the immense potential of Fine-grained sketch-based image retrieval (FG-SBIR) We propose a novel active learning sampling technique that drastically minimises the need for drawing photo sketches.
  arXiv  Detail & Related papers  (2023-09-15T20:07:14Z)
• Adaptive Fine-Grained Sketch-Based Image Retrieval [100.90633284767205]
  Recent focus on Fine-Grained Sketch-Based Image Retrieval has shifted towards generalising a model to new categories. In real-world applications, a trained FG-SBIR model is often applied to both new categories and different human sketchers. We introduce a novel model-agnostic meta-learning (MAML) based framework with several key modifications.
  arXiv  Detail & Related papers  (2022-07-04T21:07:20Z)
• Towards Unsupervised Sketch-based Image Retrieval [126.77787336692802]
  We introduce a novel framework that simultaneously performs unsupervised representation learning and sketch-photo domain alignment. Our framework achieves excellent performance in the new unsupervised setting, and performs comparably or better than state-of-the-art in the zero-shot setting.
  arXiv  Detail & Related papers  (2021-05-18T02:38:22Z)
• StyleMeUp: Towards Style-Agnostic Sketch-Based Image Retrieval [119.03470556503942]
  Crossmodal matching problem is typically solved by learning a joint embedding space where semantic content shared between photo and sketch modalities are preserved. An effective model needs to explicitly account for this style diversity, crucially, to unseen user styles. Our model can not only disentangle the cross-modal shared semantic content, but can adapt the disentanglement to any unseen user style as well, making the model truly agnostic.
  arXiv  Detail & Related papers  (2021-03-29T15:44:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.