Diagram Image Retrieval using Sketch-Based Deep Learning and Transfer Learning

• URL: http://arxiv.org/abs/2004.10780v1
• Date: Wed, 22 Apr 2020 18:27:46 GMT
• Title: Diagram Image Retrieval using Sketch-Based Deep Learning and Transfer Learning
• Authors: Manish Bhattarai, Diane Oyen, Juan Castorena, Liping Yang, Brendt Wohlberg
• Abstract summary: This paper presents a deep learning based method for image-based search for binary patent images. We begin by using deep learning to generate sketches from natural images for image retrieval. We then use our small set of manually labeled patent diagram images via transfer learning to adapt the image search from sketches of natural images to diagrams.
• Score: 28.31395148295604
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Resolution of the complex problem of image retrieval for diagram images has yet to be reached. Deep learning methods continue to excel in the fields of object detection and image classification applied to natural imagery. However, the application of such methodologies applied to binary imagery remains limited due to lack of crucial features such as textures,color and intensity information. This paper presents a deep learning based method for image-based search for binary patent images by taking advantage of existing large natural image repositories for image search and sketch-based methods (Sketches are not identical to diagrams, but they do share some characteristics; for example, both imagery types are gray scale (binary), composed of contours, and are lacking in texture). We begin by using deep learning to generate sketches from natural images for image retrieval and then train a second deep learning model on the sketches. We then use our small set of manually labeled patent diagram images via transfer learning to adapt the image search from sketches of natural images to diagrams. Our experiment results show the effectiveness of deep learning with transfer learning for detecting near-identical copies in patent images and querying similar images based on content.

Related papers

• Detecting Generated Images by Real Images Only [64.12501227493765]
  Existing generated image detection methods detect visual artifacts in generated images or learn discriminative features from both real and generated images by massive training. This paper approaches the generated image detection problem from a new perspective: Start from real images. By finding the commonality of real images and mapping them to a dense subspace in feature space, the goal is that generated images, regardless of their generative model, are then projected outside the subspace.
  arXiv  Detail & Related papers  (2023-11-02T03:09:37Z)
• Adapt and Align to Improve Zero-Shot Sketch-Based Image Retrieval [85.39613457282107]
  Cross-domain nature of sketch-based image retrieval is challenging. We present an effective Adapt and Align'' approach to address the key challenges. Inspired by recent advances in image-text foundation models (e.g., CLIP) on zero-shot scenarios, we explicitly align the learned image embedding with a more semantic text embedding to achieve the desired knowledge transfer from seen to unseen classes.
  arXiv  Detail & Related papers  (2023-05-09T03:10:15Z)
• Sketch2Saliency: Learning to Detect Salient Objects from Human Drawings [99.9788496281408]
  We study how sketches can be used as a weak label to detect salient objects present in an image. To accomplish this, we introduce a photo-to-sketch generation model that aims to generate sequential sketch coordinates corresponding to a given visual photo. Tests prove our hypothesis and delineate how our sketch-based saliency detection model gives a competitive performance compared to the state-of-the-art.
  arXiv  Detail & Related papers  (2023-03-20T23:46:46Z)
• Three-Stream Joint Network for Zero-Shot Sketch-Based Image Retrieval [15.191262439963221]
  The Zero-Shot Sketch-based Image Retrieval (ZS-SBIR) is a challenging task because of the large domain gap between sketches and natural images. We propose a novel Three-Stream Joint Training Network (3 JOIN) for the ZS-SBIR task.
  arXiv  Detail & Related papers  (2022-04-12T09:52:17Z)
• Texture for Colors: Natural Representations of Colors Using Variable Bit-Depth Textures [13.180922099929765]
  We present an automated method to transform an image to a set of binary textures that represent not only the intensities, but also the colors of the original. The system yields aesthetically pleasing binary images when tested on a variety of image sources.
  arXiv  Detail & Related papers  (2021-05-04T21:22:02Z)
• Insights From A Large-Scale Database of Material Depictions In Paintings [18.2193253052961]
  We examine the give-and-take relationship between visual recognition systems and the rich information available in the fine arts. We find that visual recognition systems designed for natural images can work surprisingly well on paintings. We show that learning from paintings can be beneficial for neural networks that are intended to be used on natural images.
  arXiv  Detail & Related papers  (2020-11-24T18:42:58Z)
• SketchEmbedNet: Learning Novel Concepts by Imitating Drawings [125.45799722437478]
  We explore properties of image representations learned by training a model to produce sketches of images. We show that this generative, class-agnostic model produces informative embeddings of images from novel examples, classes, and even novel datasets in a few-shot setting.
  arXiv  Detail & Related papers  (2020-08-27T16:43:28Z)
• Cross-Modal Hierarchical Modelling for Fine-Grained Sketch Based Image Retrieval [147.24102408745247]
  We study a further trait of sketches that has been overlooked to date, that is, they are hierarchical in terms of the levels of detail. In this paper, we design a novel network that is capable of cultivating sketch-specific hierarchies and exploiting them to match sketch with photo at corresponding hierarchical levels.
  arXiv  Detail & Related papers  (2020-07-29T20:50:25Z)
• Learning Spatial Relationships between Samples of Patent Image Shapes [14.37369942979269]
  We propose a method suitable to binary images which bridges some of the successes of deep learning (DL) The method consists on extracting the shape of interest from the binary image and applying a non-Euclidean geometric neural-net architecture to learn the local and global spatial relationships of the shape.
  arXiv  Detail & Related papers  (2020-04-12T23:05:19Z)
• Geometrically Mappable Image Features [85.81073893916414]
  Vision-based localization of an agent in a map is an important problem in robotics and computer vision. We propose a method that learns image features targeted for image-retrieval-based localization.
  arXiv  Detail & Related papers  (2020-03-21T15:36:38Z)
• Image retrieval approach based on local texture information derived from predefined patterns and spatial domain information [14.620086904601472]
  The performance of the proposed method is evaluated in terms of precision and recall on the Simplicity database. The comparative results showed that the proposed approach offers higher precision rate than many known methods.
  arXiv  Detail & Related papers  (2019-12-30T16:11:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.