Abstracting Sketches through Simple Primitives
- URL: http://arxiv.org/abs/2207.13543v1
- Date: Wed, 27 Jul 2022 14:32:39 GMT
- Title: Abstracting Sketches through Simple Primitives
- Authors: Stephan Alaniz, Massimiliano Mancini, Anjan Dutta, Diego Marcos,
Zeynep Akata
- Abstract summary: Humans show high-level of abstraction capabilities in games that require quickly communicating object information.
We propose the Primitive-based Sketch Abstraction task where the goal is to represent sketches using a fixed set of drawing primitives.
Our Primitive-Matching Network (PMN), learns interpretable abstractions of a sketch in a self supervised manner.
- Score: 53.04827416243121
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Humans show high-level of abstraction capabilities in games that require
quickly communicating object information. They decompose the message content
into multiple parts and communicate them in an interpretable protocol. Toward
equipping machines with such capabilities, we propose the Primitive-based
Sketch Abstraction task where the goal is to represent sketches using a fixed
set of drawing primitives under the influence of a budget. To solve this task,
our Primitive-Matching Network (PMN), learns interpretable abstractions of a
sketch in a self supervised manner. Specifically, PMN maps each stroke of a
sketch to its most similar primitive in a given set, predicting an affine
transformation that aligns the selected primitive to the target stroke. We
learn this stroke-to-primitive mapping end-to-end with a distance-transform
loss that is minimal when the original sketch is precisely reconstructed with
the predicted primitives. Our PMN abstraction empirically achieves the highest
performance on sketch recognition and sketch-based image retrieval given a
communication budget, while at the same time being highly interpretable. This
opens up new possibilities for sketch analysis, such as comparing sketches by
extracting the most relevant primitives that define an object category. Code is
available at https://github.com/ExplainableML/sketch-primitives.
Related papers
- SketchTriplet: Self-Supervised Scenarized Sketch-Text-Image Triplet Generation [6.39528707908268]
There continues to be a lack of large-scale paired datasets for scene sketches.
We propose a self-supervised method for scene sketch generation that does not rely on any existing scene sketch.
We contribute a large-scale dataset centered around scene sketches, comprising highly semantically consistent "text-sketch-image" triplets.
arXiv Detail & Related papers (2024-05-29T06:43:49Z) - How to Handle Sketch-Abstraction in Sketch-Based Image Retrieval? [120.49126407479717]
We propose a sketch-based image retrieval framework capable of handling sketch abstraction at varied levels.
For granularity-level abstraction understanding, we dictate that the retrieval model should not treat all abstraction-levels equally.
Our Acc.@q loss uniquely allows a sketch to narrow/broaden its focus in terms of how stringent the evaluation should be.
arXiv Detail & Related papers (2024-03-11T23:08:29Z) - Open Vocabulary Semantic Scene Sketch Understanding [5.638866331696071]
We study the underexplored but fundamental vision problem of machine understanding of freehand scene sketches.
We introduce a sketch encoder that results in semantically-aware feature space, which we evaluate by testing its performance on a semantic sketch segmentation task.
Our method outperforms zero-shot CLIP pixel accuracy of segmentation results by 37 points, reaching an accuracy of $85.5%$ on the FS-COCO sketch dataset.
arXiv Detail & Related papers (2023-12-18T19:02:07Z) - Sketch-based Video Object Segmentation: Benchmark and Analysis [55.79497833614397]
This paper introduces a new task of sketch-based video object segmentation, an associated benchmark, and a strong baseline.
Our benchmark includes three datasets, Sketch-DAVIS16, Sketch-DAVIS17 and Sketch-YouTube-VOS, which exploit human-drawn sketches as an informative yet low-cost reference for video object segmentation.
Experimental results show sketch is more effective yet annotation-efficient than other references, such as photo masks, language and scribble.
arXiv Detail & Related papers (2023-11-13T11:53:49Z) - What Can Human Sketches Do for Object Detection? [127.67444974452411]
Sketches are highly expressive, inherently capturing subjective and fine-grained visual cues.
A sketch-enabled object detection framework detects based on what textityou sketch -- textitthat zebra''
We show an intuitive synergy between foundation models (e.g., CLIP) and existing sketch models build for sketch-based image retrieval (SBIR)
In particular, we first perform independent on both sketch branches of an encoder model to build highly generalisable sketch and photo encoders.
arXiv Detail & Related papers (2023-03-27T12:33:23Z) - I Know What You Draw: Learning Grasp Detection Conditioned on a Few
Freehand Sketches [74.63313641583602]
We propose a method to generate a potential grasp configuration relevant to the sketch-depicted objects.
Our model is trained and tested in an end-to-end manner which is easy to be implemented in real-world applications.
arXiv Detail & Related papers (2022-05-09T04:23:36Z) - CLIPasso: Semantically-Aware Object Sketching [34.53644912236454]
We present an object sketching method that can achieve different levels of abstraction, guided by geometric and semantic simplifications.
We define a sketch as a set of B'ezier curves and use a differentiizer to optimize the parameters of the curves directly with respect to a CLIP-based perceptual loss.
arXiv Detail & Related papers (2022-02-11T18:35:25Z) - SketchLattice: Latticed Representation for Sketch Manipulation [30.092468954557468]
Key challenge in designing a sketch representation lies with handling the abstract and iconic nature of sketches.
We propose a lattice structured sketch representation that not only removes the bottleneck of requiring vector data but also preserves the structural cues that vector data provides.
Our lattice representation could be effectively encoded using a graph model, that uses significantly fewer model parameters (13.5 times lesser) than existing state-of-the-art.
arXiv Detail & Related papers (2021-08-26T08:02:21Z) - Cross-Modal Hierarchical Modelling for Fine-Grained Sketch Based Image
Retrieval [147.24102408745247]
We study a further trait of sketches that has been overlooked to date, that is, they are hierarchical in terms of the levels of detail.
In this paper, we design a novel network that is capable of cultivating sketch-specific hierarchies and exploiting them to match sketch with photo at corresponding hierarchical levels.
arXiv Detail & Related papers (2020-07-29T20:50:25Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.