Enhance Sketch Recognition's Explainability via Semantic Component-Level Parsing

• URL: http://arxiv.org/abs/2312.07875v1
• Date: Wed, 13 Dec 2023 03:33:16 GMT
• Title: Enhance Sketch Recognition's Explainability via Semantic Component-Level Parsing
• Abstract summary: A semantic component-level memory module is constructed and embedded in the proposed structured sketch recognition network. The memory keys representing semantic components of each sketch category can be self-learned and enhance the recognition network's explainability. Experiments on the SPG and SketchIME datasets demonstrate the memory module's flexibility and the recognition network's explainability.
• Score: 10.76791271745353
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Free-hand sketches are appealing for humans as a universal tool to depict the visual world. Humans can recognize varied sketches of a category easily by identifying the concurrence and layout of the intrinsic semantic components of the category, since humans draw free-hand sketches based a common consensus that which types of semantic components constitute each sketch category. For example, an airplane should at least have a fuselage and wings. Based on this analysis, a semantic component-level memory module is constructed and embedded in the proposed structured sketch recognition network in this paper. The memory keys representing semantic components of each sketch category can be self-learned and enhance the recognition network's explainability. Our proposed networks can deal with different situations of sketch recognition, i.e., with or without semantic components labels of strokes. Experiments on the SPG and SketchIME datasets demonstrate the memory module's flexibility and the recognition network's explainability. The code and data are available at https://github.com/GuangmingZhu/SketchESC.

Related papers

• SEA: Evaluating Sketch Abstraction Efficiency via Element-level Commonsense Visual Question Answering [4.267729144203295]
  We introduce SEA (Sketch Evaluation metric for Abstraction efficiency), a reference-free metric that assesses how economically a sketch represents class-defining visual elements.<n>We present CommonSketch, the first semantically annotated sketch dataset, comprising 23,100 human-drawn sketches across 300 classes.
  arXiv  Detail & Related papers  (2026-03-30T12:30:51Z)
• Sketch-based Video Object Segmentation: Benchmark and Analysis [55.79497833614397]
  This paper introduces a new task of sketch-based video object segmentation, an associated benchmark, and a strong baseline. Our benchmark includes three datasets, Sketch-DAVIS16, Sketch-DAVIS17 and Sketch-YouTube-VOS, which exploit human-drawn sketches as an informative yet low-cost reference for video object segmentation. Experimental results show sketch is more effective yet annotation-efficient than other references, such as photo masks, language and scribble.
  arXiv  Detail & Related papers  (2023-11-13T11:53:49Z)
• Learning Dense Correspondences between Photos and Sketches [6.2420740599184175]
  Humans effortlessly grasp the connection between sketches and real-world objects, even when these sketches are far from realistic. We introduce a new sketch-photo correspondence benchmark, $textitPSC6k$, containing 150K annotations of 6250 sketch-photo pairs across 125 object categories. Second, we propose a self-supervised method for learning dense correspondences between sketch-photo pairs.
  arXiv  Detail & Related papers  (2023-07-24T17:45:40Z)
• SketchXAI: A First Look at Explainability for Human Sketches [104.13322289903577]
  This paper introduces human sketches to the landscape of XAI (Explainable Artificial Intelligence) We argue that sketch as a human-centred'' data form, represents a natural interface to study explainability. We design a sketch encoder that accommodates the intrinsic properties of strokes: shape, location, and order.
  arXiv  Detail & Related papers  (2023-04-23T20:28:38Z)
• Abstracting Sketches through Simple Primitives [53.04827416243121]
  Humans show high-level of abstraction capabilities in games that require quickly communicating object information. We propose the Primitive-based Sketch Abstraction task where the goal is to represent sketches using a fixed set of drawing primitives. Our Primitive-Matching Network (PMN), learns interpretable abstractions of a sketch in a self supervised manner.
  arXiv  Detail & Related papers  (2022-07-27T14:32:39Z)
• I Know What You Draw: Learning Grasp Detection Conditioned on a Few Freehand Sketches [74.63313641583602]
  We propose a method to generate a potential grasp configuration relevant to the sketch-depicted objects. Our model is trained and tested in an end-to-end manner which is easy to be implemented in real-world applications.
  arXiv  Detail & Related papers  (2022-05-09T04:23:36Z)
• CLIPasso: Semantically-Aware Object Sketching [34.53644912236454]
  We present an object sketching method that can achieve different levels of abstraction, guided by geometric and semantic simplifications. We define a sketch as a set of B'ezier curves and use a differentiizer to optimize the parameters of the curves directly with respect to a CLIP-based perceptual loss.
  arXiv  Detail & Related papers  (2022-02-11T18:35:25Z)
• One Sketch for All: One-Shot Personalized Sketch Segmentation [84.45203849671003]
  We present the first one-shot personalized sketch segmentation method. We aim to segment all sketches belonging to the same category with a single sketch with a given part annotation. We preserve the parts semantics embedded in the exemplar, and we are robust to input style and abstraction.
  arXiv  Detail & Related papers  (2021-12-20T20:10:44Z)
• Cross-Modal Hierarchical Modelling for Fine-Grained Sketch Based Image Retrieval [147.24102408745247]
  We study a further trait of sketches that has been overlooked to date, that is, they are hierarchical in terms of the levels of detail. In this paper, we design a novel network that is capable of cultivating sketch-specific hierarchies and exploiting them to match sketch with photo at corresponding hierarchical levels.
  arXiv  Detail & Related papers  (2020-07-29T20:50:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.