Instance Segmentation of Scene Sketches Using Natural Image Priors

• URL: http://arxiv.org/abs/2502.09608v1
• Date: Thu, 13 Feb 2025 18:56:05 GMT
• Title: Instance Segmentation of Scene Sketches Using Natural Image Priors
• Authors: Mia Tang, Yael Vinker, Chuan Yan, Lvmin Zhang, Maneesh Agrawala,
• Abstract summary: We introduce SketchSeg, a method for instance segmentation of scene sketches. Our approach adapts state-of-the-art image segmentation and object detection models to the sketch domain by employing class-agnostic fine-tuning. Our method organizes sketches into sorted layers, where occluded instances are inpainted, enabling advanced sketch editing applications.
• Score: 30.518717641778753
• License:
• Abstract: Sketch segmentation involves grouping pixels within a sketch that belong to the same object or instance. It serves as a valuable tool for sketch editing tasks, such as moving, scaling, or removing specific components. While image segmentation models have demonstrated remarkable capabilities in recent years, sketches present unique challenges for these models due to their sparse nature and wide variation in styles. We introduce SketchSeg, a method for instance segmentation of raster scene sketches. Our approach adapts state-of-the-art image segmentation and object detection models to the sketch domain by employing class-agnostic fine-tuning and refining segmentation masks using depth cues. Furthermore, our method organizes sketches into sorted layers, where occluded instances are inpainted, enabling advanced sketch editing applications. As existing datasets in this domain lack variation in sketch styles, we construct a synthetic scene sketch segmentation dataset featuring sketches with diverse brush strokes and varying levels of detail. We use this dataset to demonstrate the robustness of our approach and will release it to promote further research in the field. Project webpage: https://sketchseg.github.io/sketch-seg/

Related papers

• Freestyle Sketch-in-the-Loop Image Segmentation [116.1810651297801]
  We introduce a "sketch-in-the-loop" image segmentation framework, enabling the segmentation of visual concepts partially, completely, or in groupings. This framework capitalises on the synergy between sketch-based image retrieval models and large-scale pre-trained models. Our purpose-made augmentation strategy enhances the versatility of our sketch-guided mask generation, allowing segmentation at multiple levels.
  arXiv  Detail & Related papers  (2025-01-27T13:07:51Z)
• Class-Agnostic Visio-Temporal Scene Sketch Semantic Segmentation [0.9208007322096532]
  Scene sketch semantic segmentation is a crucial task for various applications including sketch-to-image retrieval and scene understanding. Existing sketch segmentation methods treat sketches as bitmap images, leading to the loss of temporal order among strokes. We propose a Class-Agnostic-Temporal Network (CAVT) for scene sketch semantic segmentation.
  arXiv  Detail & Related papers  (2024-09-30T22:34:29Z)
• SketchTriplet: Self-Supervised Scenarized Sketch-Text-Image Triplet Generation [6.39528707908268]
  There continues to be a lack of large-scale paired datasets for scene sketches. We propose a self-supervised method for scene sketch generation that does not rely on any existing scene sketch. We contribute a large-scale dataset centered around scene sketches, comprising highly semantically consistent "text-sketch-image" triplets.
  arXiv  Detail & Related papers  (2024-05-29T06:43:49Z)
• Sketch-based Video Object Segmentation: Benchmark and Analysis [55.79497833614397]
  This paper introduces a new task of sketch-based video object segmentation, an associated benchmark, and a strong baseline. Our benchmark includes three datasets, Sketch-DAVIS16, Sketch-DAVIS17 and Sketch-YouTube-VOS, which exploit human-drawn sketches as an informative yet low-cost reference for video object segmentation. Experimental results show sketch is more effective yet annotation-efficient than other references, such as photo masks, language and scribble.
  arXiv  Detail & Related papers  (2023-11-13T11:53:49Z)
• SketchFFusion: Sketch-guided image editing with diffusion model [25.63913085329606]
  Sketch-guided image editing aims to achieve local fine-tuning of the image based on the sketch information provided by the user. We propose a sketch generation scheme that can preserve the main contours of an image and closely adhere to the actual sketch style drawn by the user.
  arXiv  Detail & Related papers  (2023-04-06T15:54:18Z)
• Query-guided Attention in Vision Transformers for Localizing Objects Using a Single Sketch [17.63475613154152]
  Given a crude hand-drawn sketch of an object, the goal is to localize all instances of the same object on the target image. This problem proves difficult due to the abstract nature of hand-drawn sketches, variations in the style and quality of sketches, and the large domain gap existing between the sketches and the natural images. We propose a sketch-guided vision transformer encoder that uses cross-attention after each block of the transformer-based image encoder to learn query-conditioned image features.
  arXiv  Detail & Related papers  (2023-03-15T17:26:17Z)
• One Sketch for All: One-Shot Personalized Sketch Segmentation [84.45203849671003]
  We present the first one-shot personalized sketch segmentation method. We aim to segment all sketches belonging to the same category with a single sketch with a given part annotation. We preserve the parts semantics embedded in the exemplar, and we are robust to input style and abstraction.
  arXiv  Detail & Related papers  (2021-12-20T20:10:44Z)
• SketchEdit: Mask-Free Local Image Manipulation with Partial Sketches [95.45728042499836]
  We propose a new paradigm of sketch-based image manipulation: mask-free local image manipulation. Our model automatically predicts the target modification region and encodes it into a structure style vector. A generator then synthesizes the new image content based on the style vector and sketch.
  arXiv  Detail & Related papers  (2021-11-30T02:42:31Z)
• SketchDesc: Learning Local Sketch Descriptors for Multi-view Correspondence [68.63311821718416]
  We study the problem of multi-view sketch correspondence, where we take as input multiple freehand sketches with different views of the same object. This problem is challenging since the visual features of corresponding points at different views can be very different. We take a deep learning approach and learn a novel local sketch descriptor from data.
  arXiv  Detail & Related papers  (2020-01-16T11:31:21Z)
• Deep Plastic Surgery: Robust and Controllable Image Editing with Human-Drawn Sketches [133.01690754567252]
  Sketch-based image editing aims to synthesize and modify photos based on the structural information provided by the human-drawn sketches. Deep Plastic Surgery is a novel, robust and controllable image editing framework that allows users to interactively edit images using hand-drawn sketch inputs.
  arXiv  Detail & Related papers  (2020-01-09T08:57:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.