SEVA: Leveraging sketches to evaluate alignment between human and machine visual abstraction

• URL: http://arxiv.org/abs/2312.03035v1
• Date: Tue, 5 Dec 2023 13:54:55 GMT
• Title: SEVA: Leveraging sketches to evaluate alignment between human and machine visual abstraction
• Authors: Kushin Mukherjee and Holly Huey and Xuanchen Lu and Yael Vinker and Rio Aguina-Kang and Ariel Shamir and Judith E. Fan
• Abstract summary: Sketching is a powerful tool for creating abstract images that are sparse but meaningful. Current vision algorithms have achieved high performance on a variety of visual tasks. It remains unclear to what extent they understand sketches in a human-like way.
• Score: 19.70530050403922
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Sketching is a powerful tool for creating abstract images that are sparse but meaningful. Sketch understanding poses fundamental challenges for general-purpose vision algorithms because it requires robustness to the sparsity of sketches relative to natural visual inputs and because it demands tolerance for semantic ambiguity, as sketches can reliably evoke multiple meanings. While current vision algorithms have achieved high performance on a variety of visual tasks, it remains unclear to what extent they understand sketches in a human-like way. Here we introduce SEVA, a new benchmark dataset containing approximately 90K human-generated sketches of 128 object concepts produced under different time constraints, and thus systematically varying in sparsity. We evaluated a suite of state-of-the-art vision algorithms on their ability to correctly identify the target concept depicted in these sketches and to generate responses that are strongly aligned with human response patterns on the same sketch recognition task. We found that vision algorithms that better predicted human sketch recognition performance also better approximated human uncertainty about sketch meaning, but there remains a sizable gap between model and human response patterns. To explore the potential of models that emulate human visual abstraction in generative tasks, we conducted further evaluations of a recently developed sketch generation algorithm (Vinker et al., 2022) capable of generating sketches that vary in sparsity. We hope that public release of this dataset and evaluation protocol will catalyze progress towards algorithms with enhanced capacities for human-like visual abstraction.

Related papers

• HINT: Learning Complete Human Neural Representations from Limited Viewpoints [69.76947323932107]
  We propose a NeRF-based algorithm able to learn a detailed and complete human model from limited viewing angles. As a result, our method can reconstruct complete humans even from a few viewing angles, increasing performance by more than 15% PSNR.
  arXiv  Detail & Related papers  (2024-05-30T05:43:09Z)
• Explaining Deep Face Algorithms through Visualization: A Survey [57.60696799018538]
  This work undertakes a first-of-its-kind meta-analysis of explainability algorithms in the face domain. We review existing face explainability works and reveal valuable insights into the structure and hierarchy of face networks.
  arXiv  Detail & Related papers  (2023-09-26T07:16:39Z)
• Sketch2Saliency: Learning to Detect Salient Objects from Human Drawings [99.9788496281408]
  We study how sketches can be used as a weak label to detect salient objects present in an image. To accomplish this, we introduce a photo-to-sketch generation model that aims to generate sequential sketch coordinates corresponding to a given visual photo. Tests prove our hypothesis and delineate how our sketch-based saliency detection model gives a competitive performance compared to the state-of-the-art.
  arXiv  Detail & Related papers  (2023-03-20T23:46:46Z)
• Abstracting Sketches through Simple Primitives [53.04827416243121]
  Humans show high-level of abstraction capabilities in games that require quickly communicating object information. We propose the Primitive-based Sketch Abstraction task where the goal is to represent sketches using a fixed set of drawing primitives. Our Primitive-Matching Network (PMN), learns interpretable abstractions of a sketch in a self supervised manner.
  arXiv  Detail & Related papers  (2022-07-27T14:32:39Z)
• I Know What You Draw: Learning Grasp Detection Conditioned on a Few Freehand Sketches [74.63313641583602]
  We propose a method to generate a potential grasp configuration relevant to the sketch-depicted objects. Our model is trained and tested in an end-to-end manner which is easy to be implemented in real-world applications.
  arXiv  Detail & Related papers  (2022-05-09T04:23:36Z)
• Behind the Machine's Gaze: Biologically Constrained Neural Networks Exhibit Human-like Visual Attention [40.878963450471026]
  We propose the Neural Visual Attention (NeVA) algorithm to generate visual scanpaths in a top-down manner. We show that the proposed method outperforms state-of-the-art unsupervised human attention models in terms of similarity to human scanpaths.
  arXiv  Detail & Related papers  (2022-04-19T18:57:47Z)
• Sketch2Model: View-Aware 3D Modeling from Single Free-Hand Sketches [4.781615891172263]
  We investigate the problem of generating 3D meshes from single free-hand sketches, aiming at fast 3D modeling for novice users. We address the importance of viewpoint specification for overcoming ambiguities, and propose a novel view-aware generation approach.
  arXiv  Detail & Related papers  (2021-05-14T06:27:48Z)
• On Learning Semantic Representations for Million-Scale Free-Hand Sketches [146.52892067335128]
  We study learning semantic representations for million-scale free-hand sketches. We propose a dual-branch CNNRNN network architecture to represent sketches. We explore learning the sketch-oriented semantic representations in hashing retrieval and zero-shot recognition.
  arXiv  Detail & Related papers  (2020-07-07T15:23:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.