Sketchformer: Transformer-based Representation for Sketched Structure

• URL: http://arxiv.org/abs/2002.10381v1
• Date: Mon, 24 Feb 2020 17:11:53 GMT
• Title: Sketchformer: Transformer-based Representation for Sketched Structure
• Authors: Leo Sampaio Ferraz Ribeiro, Tu Bui, John Collomosse, Moacir Ponti
• Abstract summary: Sketchformer is a transformer-based representation for encoding free-hand sketches input in a vector form. We report several variants exploring continuous and tokenized input representations, and contrast their performance. Our learned embedding, driven by a dictionary learning tokenization scheme, yields state of the art performance in classification and image retrieval tasks.
• Score: 12.448155157592895
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Sketchformer is a novel transformer-based representation for encoding free-hand sketches input in a vector form, i.e. as a sequence of strokes. Sketchformer effectively addresses multiple tasks: sketch classification, sketch based image retrieval (SBIR), and the reconstruction and interpolation of sketches. We report several variants exploring continuous and tokenized input representations, and contrast their performance. Our learned embedding, driven by a dictionary learning tokenization scheme, yields state of the art performance in classification and image retrieval tasks, when compared against baseline representations driven by LSTM sequence to sequence architectures: SketchRNN and derivatives. We show that sketch reconstruction and interpolation are improved significantly by the Sketchformer embedding for complex sketches with longer stroke sequences.

Related papers

• SketchGPT: Autoregressive Modeling for Sketch Generation and Recognition [4.6519578789100215]
  SketchGPT is a flexible framework that employs a sequence-to-sequence autoregressive model for sketch generation, and completion. By mapping complex sketches into simplified sequences of abstract primitives, our approach significantly streamlines the input for autoregressive modeling.
  arXiv  Detail & Related papers  (2024-05-06T01:24:14Z)
• Equipping Sketch Patches with Context-Aware Positional Encoding for Graphic Sketch Representation [4.961362040453441]
  We propose a variant-drawing-protected method for learning graphic sketch representation. Instead of injecting sketch drawings into graph edges, we embed these sequential information into graph nodes only. Experimental results indicate that our method significantly improves sketch healing and controllable sketch synthesis.
  arXiv  Detail & Related papers  (2024-03-26T09:26:12Z)
• SketchINR: A First Look into Sketches as Implicit Neural Representations [120.4152701687737]
  We propose SketchINR, to advance the representation of vector sketches with implicit neural models. A variable length vector sketch is compressed into a latent space of fixed dimension that implicitly encodes the underlying shape as a function of time and strokes. For the first time, SketchINR emulates the human ability to reproduce a sketch with varying abstraction in terms of number and complexity of strokes.
  arXiv  Detail & Related papers  (2024-03-14T12:49:29Z)
• SENS: Part-Aware Sketch-based Implicit Neural Shape Modeling [124.3266213819203]
  We present SENS, a novel method for generating and editing 3D models from hand-drawn sketches. S SENS analyzes the sketch and encodes its parts into ViT patch encoding. S SENS supports refinement via part reconstruction, allowing for nuanced adjustments and artifact removal.
  arXiv  Detail & Related papers  (2023-06-09T17:50:53Z)
• Abstracting Sketches through Simple Primitives [53.04827416243121]
  Humans show high-level of abstraction capabilities in games that require quickly communicating object information. We propose the Primitive-based Sketch Abstraction task where the goal is to represent sketches using a fixed set of drawing primitives. Our Primitive-Matching Network (PMN), learns interpretable abstractions of a sketch in a self supervised manner.
  arXiv  Detail & Related papers  (2022-07-27T14:32:39Z)
• SSR-GNNs: Stroke-based Sketch Representation with Graph Neural Networks [34.759306840182205]
  This paper investigates a graph representation for sketches, where the information of strokes, i.e., parts of a sketch, are encoded on vertices and information of inter-stroke on edges. The resultant graph representation facilitates the training of a Graph Neural Networks for classification tasks. The proposed representation enables generation of novel sketches that are structurally similar to while separable from the existing dataset.
  arXiv  Detail & Related papers  (2022-04-27T19:18:01Z)
• SketchLattice: Latticed Representation for Sketch Manipulation [30.092468954557468]
  Key challenge in designing a sketch representation lies with handling the abstract and iconic nature of sketches. We propose a lattice structured sketch representation that not only removes the bottleneck of requiring vector data but also preserves the structural cues that vector data provides. Our lattice representation could be effectively encoded using a graph model, that uses significantly fewer model parameters (13.5 times lesser) than existing state-of-the-art.
  arXiv  Detail & Related papers  (2021-08-26T08:02:21Z)
• Compositional Sketch Search [91.84489055347585]
  We present an algorithm for searching image collections using free-hand sketches. We exploit drawings as a concise and intuitive representation for specifying entire scene compositions.
  arXiv  Detail & Related papers  (2021-06-15T09:38:09Z)
• Sketch-BERT: Learning Sketch Bidirectional Encoder Representation from Transformers by Self-supervised Learning of Sketch Gestalt [125.17887147597567]
  We present a model of learning Sketch BiBERT Representation from Transformer (Sketch-BERT) We generalize BERT to sketch domain, with the novel proposed components and pre-training algorithms. We show that the learned representation of Sketch-BERT can help and improve the performance of the downstream tasks of sketch recognition, sketch retrieval, and sketch gestalt.
  arXiv  Detail & Related papers  (2020-05-19T01:35:44Z)
• SketchDesc: Learning Local Sketch Descriptors for Multi-view Correspondence [68.63311821718416]
  We study the problem of multi-view sketch correspondence, where we take as input multiple freehand sketches with different views of the same object. This problem is challenging since the visual features of corresponding points at different views can be very different. We take a deep learning approach and learn a novel local sketch descriptor from data.
  arXiv  Detail & Related papers  (2020-01-16T11:31:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.