Deep Self-Supervised Representation Learning for Free-Hand Sketch

• URL: http://arxiv.org/abs/2002.00867v1
• Date: Mon, 3 Feb 2020 16:28:29 GMT
• Title: Deep Self-Supervised Representation Learning for Free-Hand Sketch
• Authors: Peng Xu, Zeyu Song, Qiyue Yin, Yi-Zhe Song, Liang Wang
• Abstract summary: We tackle the problem of self-supervised representation learning for free-hand sketches. Key for the success of our self-supervised learning paradigm lies with our sketch-specific designs. We show that the proposed approach outperforms the state-of-the-art unsupervised representation learning methods.
• Score: 51.101565480583304
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper, we tackle for the first time, the problem of self-supervised representation learning for free-hand sketches. This importantly addresses a common problem faced by the sketch community -- that annotated supervisory data are difficult to obtain. This problem is very challenging in that sketches are highly abstract and subject to different drawing styles, making existing solutions tailored for photos unsuitable. Key for the success of our self-supervised learning paradigm lies with our sketch-specific designs: (i) we propose a set of pretext tasks specifically designed for sketches that mimic different drawing styles, and (ii) we further exploit the use of a textual convolution network (TCN) in a dual-branch architecture for sketch feature learning, as means to accommodate the sequential stroke nature of sketches. We demonstrate the superiority of our sketch-specific designs through two sketch-related applications (retrieval and recognition) on a million-scale sketch dataset, and show that the proposed approach outperforms the state-of-the-art unsupervised representation learning methods, and significantly narrows the performance gap between with supervised representation learning.

Related papers

• CustomSketching: Sketch Concept Extraction for Sketch-based Image Synthesis and Editing [21.12815542848095]
  Personalization techniques for large text-to-image (T2I) models allow users to incorporate new concepts from reference images. Existing methods primarily rely on textual descriptions, leading to limited control over customized images. We identify sketches as an intuitive and versatile representation that can facilitate such control.
  arXiv  Detail & Related papers  (2024-02-27T15:52:59Z)
• Sketch2Saliency: Learning to Detect Salient Objects from Human Drawings [99.9788496281408]
  We study how sketches can be used as a weak label to detect salient objects present in an image. To accomplish this, we introduce a photo-to-sketch generation model that aims to generate sequential sketch coordinates corresponding to a given visual photo. Tests prove our hypothesis and delineate how our sketch-based saliency detection model gives a competitive performance compared to the state-of-the-art.
  arXiv  Detail & Related papers  (2023-03-20T23:46:46Z)
• I Know What You Draw: Learning Grasp Detection Conditioned on a Few Freehand Sketches [74.63313641583602]
  We propose a method to generate a potential grasp configuration relevant to the sketch-depicted objects. Our model is trained and tested in an end-to-end manner which is easy to be implemented in real-world applications.
  arXiv  Detail & Related papers  (2022-05-09T04:23:36Z)
• FS-COCO: Towards Understanding of Freehand Sketches of Common Objects in Context [112.07988211268612]
  We advance sketch research to scenes with the first dataset of freehand scene sketches, FS-COCO. Our dataset comprises 10,000 freehand scene vector sketches with per point space-time information by 100 non-expert individuals. We study for the first time the problem of the fine-grained image retrieval from freehand scene sketches and sketch captions.
  arXiv  Detail & Related papers  (2022-03-04T03:00:51Z)
• Multi-granularity Association Learning Framework for on-the-fly Fine-Grained Sketch-based Image Retrieval [7.797006835701767]
  Fine-grained sketch-based image retrieval (FG-SBIR) addresses the problem of retrieving a particular photo in a given query sketch. In this study, we aim to retrieve the target photo with the least number of strokes possible (incomplete sketch) We propose a multi-granularity association learning framework that further optimize the embedding space of all incomplete sketches.
  arXiv  Detail & Related papers  (2022-01-13T14:38:50Z)
• On Learning Semantic Representations for Million-Scale Free-Hand Sketches [146.52892067335128]
  We study learning semantic representations for million-scale free-hand sketches. We propose a dual-branch CNNRNN network architecture to represent sketches. We explore learning the sketch-oriented semantic representations in hashing retrieval and zero-shot recognition.
  arXiv  Detail & Related papers  (2020-07-07T15:23:22Z)
• SketchDesc: Learning Local Sketch Descriptors for Multi-view Correspondence [68.63311821718416]
  We study the problem of multi-view sketch correspondence, where we take as input multiple freehand sketches with different views of the same object. This problem is challenging since the visual features of corresponding points at different views can be very different. We take a deep learning approach and learn a novel local sketch descriptor from data.
  arXiv  Detail & Related papers  (2020-01-16T11:31:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.