Multi-Granularity Representation Learning for Sketch-based Dynamic Face Image Retrieval

• URL: http://arxiv.org/abs/2401.00371v1
• Date: Sun, 31 Dec 2023 02:17:14 GMT
• Title: Multi-Granularity Representation Learning for Sketch-based Dynamic Face Image Retrieval
• Authors: Liang Wang, Dawei Dai, Shiyu Fu, Guoyin Wang
• Abstract summary: In specific scenarios, face sketch can be used to identify a person. Sketch less face image retrieval (SLFIR) attempts to overcome the barriers by providing a means for humans and machines to interact during the drawing process. In this study, we propose a multigranularity (MG) representation learning (MGRL) method to address the SLFIR problem.
• Score: 13.287197931943961
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In specific scenarios, face sketch can be used to identify a person. However, drawing a face sketch often requires exceptional skill and is time-consuming, limiting its widespread applications in actual scenarios. The new framework of sketch less face image retrieval (SLFIR)[1] attempts to overcome the barriers by providing a means for humans and machines to interact during the drawing process. Considering SLFIR problem, there is a large gap between a partial sketch with few strokes and any whole face photo, resulting in poor performance at the early stages. In this study, we propose a multigranularity (MG) representation learning (MGRL) method to address the SLFIR problem, in which we learn the representation of different granularity regions for a partial sketch, and then, by combining all MG regions of the sketches and images, the final distance was determined. In the experiments, our method outperformed state-of-the-art baselines in terms of early retrieval on two accessible datasets. Codes are available at https://github.com/ddw2AIGROUP2CQUPT/MGRL.

Related papers

• Stylized Face Sketch Extraction via Generative Prior with Limited Data [6.727433982111717]
  StyleSketch is a method for extracting high-resolution stylized sketches from a face image. Using the rich semantics of the deep features from a pretrained StyleGAN, we are able to train a sketch generator with 16 pairs of face and the corresponding sketch images.
  arXiv  Detail & Related papers  (2024-03-17T16:25:25Z)
• Towards Interactive Image Inpainting via Sketch Refinement [13.34066589008464]
  We propose a two-stage image inpainting method termed SketchRefiner. In the first stage, we propose using a cross-correlation loss function to robustly calibrate and refine the user-provided sketches. In the second stage, we learn to extract informative features from the abstracted sketches in the feature space and modulate the inpainting process.
  arXiv  Detail & Related papers  (2023-06-01T07:15:54Z)
• Picture that Sketch: Photorealistic Image Generation from Abstract Sketches [109.69076457732632]
  Given an abstract, deformed, ordinary sketch from untrained amateurs like you and me, this paper turns it into a photorealistic image. We do not dictate an edgemap-like sketch to start with, but aim to work with abstract free-hand human sketches. In doing so, we essentially democratise the sketch-to-photo pipeline, "picturing" a sketch regardless of how good you sketch.
  arXiv  Detail & Related papers  (2023-03-20T14:49:03Z)
• Sketch Less Face Image Retrieval: A New Challenge [9.703239229149261]
  Drawing a complete face sketch often needs skills and takes time, which hinders its widespread applicability in the practice. In this study, we proposed sketch less face image retrieval (SLFIR), in which the retrieval was carried out at each stroke and aim to retrieve the target face photo using a partial sketch with as few strokes as possible. Experiments indicate that the new framework can finish the retrieval using a partial or pool drawing sketch.
  arXiv  Detail & Related papers  (2023-02-11T02:36:00Z)
• Multi-granularity Association Learning Framework for on-the-fly Fine-Grained Sketch-based Image Retrieval [7.797006835701767]
  Fine-grained sketch-based image retrieval (FG-SBIR) addresses the problem of retrieving a particular photo in a given query sketch. In this study, we aim to retrieve the target photo with the least number of strokes possible (incomplete sketch) We propose a multi-granularity association learning framework that further optimize the embedding space of all incomplete sketches.
  arXiv  Detail & Related papers  (2022-01-13T14:38:50Z)
• Fully Context-Aware Image Inpainting with a Learned Semantic Pyramid [102.24539566851809]
  Restoring reasonable and realistic content for arbitrary missing regions in images is an important yet challenging task. Recent image inpainting models have made significant progress in generating vivid visual details, but they can still lead to texture blurring or structural distortions. We propose the Semantic Pyramid Network (SPN) motivated by the idea that learning multi-scale semantic priors can greatly benefit the recovery of locally missing content in images.
  arXiv  Detail & Related papers  (2021-12-08T04:33:33Z)
• Domain-Smoothing Network for Zero-Shot Sketch-Based Image Retrieval [66.37346493506737]
  Zero-Shot Sketch-Based Image Retrieval (ZS-SBIR) is a novel cross-modal retrieval task. We propose a novel Domain-Smoothing Network (DSN) for ZS-SBIR. Our approach notably outperforms the state-of-the-art methods in both Sketchy and TU-Berlin datasets.
  arXiv  Detail & Related papers  (2021-06-22T14:58:08Z)
• DeepFacePencil: Creating Face Images from Freehand Sketches [77.00929179469559]
  Existing image-to-image translation methods require a large-scale dataset of paired sketches and images for supervision. We propose DeepFacePencil, an effective tool that is able to generate photo-realistic face images from hand-drawn sketches.
  arXiv  Detail & Related papers  (2020-08-31T03:35:21Z)
• Cross-Modal Hierarchical Modelling for Fine-Grained Sketch Based Image Retrieval [147.24102408745247]
  We study a further trait of sketches that has been overlooked to date, that is, they are hierarchical in terms of the levels of detail. In this paper, we design a novel network that is capable of cultivating sketch-specific hierarchies and exploiting them to match sketch with photo at corresponding hierarchical levels.
  arXiv  Detail & Related papers  (2020-07-29T20:50:25Z)
• Sketch Less for More: On-the-Fly Fine-Grained Sketch Based Image Retrieval [203.2520862597357]
  Fine-grained sketch-based image retrieval (FG-SBIR) addresses the problem of retrieving a particular photo instance given a user's query sketch. We reformulate the conventional FG-SBIR framework to tackle these challenges. We propose an on-the-fly design that starts retrieving as soon as the user starts drawing.
  arXiv  Detail & Related papers  (2020-02-24T15:36:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.