Face Sketch Synthesis via Semantic-Driven Generative Adversarial Network

• URL: http://arxiv.org/abs/2106.15121v1
• Date: Tue, 29 Jun 2021 07:03:56 GMT
• Title: Face Sketch Synthesis via Semantic-Driven Generative Adversarial Network
• Authors: Xingqun Qi, Muyi Sun, Weining Wang, Xiaoxiao Dong, Qi Li, Caifeng Shan
• Abstract summary: We propose a novel Semantic-Driven Generative Adrial Network (SDGAN) which embeds global structure-level style injection and local class-level knowledge re-weighting. Specifically, we conduct facial saliency detection on the input face photos to provide overall facial texture structure. In addition, we exploit face parsing layouts as the semantic-level spatial prior to enforce globally structural style injection in the generator of SDGAN.
• Score: 10.226808267718523
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Face sketch synthesis has made significant progress with the development of deep neural networks in these years. The delicate depiction of sketch portraits facilitates a wide range of applications like digital entertainment and law enforcement. However, accurate and realistic face sketch generation is still a challenging task due to the illumination variations and complex backgrounds in the real scenes. To tackle these challenges, we propose a novel Semantic-Driven Generative Adversarial Network (SDGAN) which embeds global structure-level style injection and local class-level knowledge re-weighting. Specifically, we conduct facial saliency detection on the input face photos to provide overall facial texture structure, which could be used as a global type of prior information. In addition, we exploit face parsing layouts as the semantic-level spatial prior to enforce globally structural style injection in the generator of SDGAN. Furthermore, to enhance the realistic effect of the details, we propose a novel Adaptive Re-weighting Loss (ARLoss) which dedicates to balance the contributions of different semantic classes. Experimentally, our extensive experiments on CUFS and CUFSF datasets show that our proposed algorithm achieves state-of-the-art performance.

Related papers

• MFCLIP: Multi-modal Fine-grained CLIP for Generalizable Diffusion Face Forgery Detection [64.29452783056253]
  The rapid development of photo-realistic face generation methods has raised significant concerns in society and academia. Although existing approaches mainly capture face forgery patterns using image modality, other modalities like fine-grained noises and texts are not fully explored. We propose a novel multi-modal fine-grained CLIP (MFCLIP) model, which mines comprehensive and fine-grained forgery traces across image-noise modalities.
  arXiv  Detail & Related papers  (2024-09-15T13:08:59Z)
• Learning Position-Aware Implicit Neural Network for Real-World Face Inpainting [55.87303287274932]
  Face inpainting requires the model to have a precise global understanding of the facial position structure. In this paper, we propose an textbfImplicit textbfNeural textbfInpainting textbfNetwork (IN$2$) to handle arbitrary-shape face images in real-world scenarios.
  arXiv  Detail & Related papers  (2024-01-19T07:31:44Z)
• DeepPortraitDrawing: Generating Human Body Images from Freehand Sketches [75.4318318890065]
  We present DeepDrawing, a framework for converting roughly drawn sketches to realistic human body images. To encode complicated body shapes under various poses, we take a local-to-global approach. Our method produces more realistic images than the state-of-the-art sketch-to-image synthesis techniques.
  arXiv  Detail & Related papers  (2022-05-04T14:02:45Z)
• MOST-Net: A Memory Oriented Style Transfer Network for Face Sketch Synthesis [41.80739104463557]
  Face sketch synthesis has been widely used in multi-media entertainment and law enforcement. Current image-to-image translation-based face sketch synthesis frequently encounters over-fitting problems when it comes to small-scale datasets. We present an end-to-end Memory Oriented Style Transfer Network (MOST-Net) for face sketch synthesis which can produce high-fidelity sketches with limited data.
  arXiv  Detail & Related papers  (2022-02-08T01:51:24Z)
• Realistic Full-Body Anonymization with Surface-Guided GANs [7.37907896341367]
  We propose a new anonymization method that generates realistic humans for in-the-wild images. A key part of our design is to guide adversarial nets by dense pixel-to-surface correspondences between an image and a canonical 3D surface. We demonstrate that surface guidance significantly improves image quality and diversity of samples, yielding a highly practical generator.
  arXiv  Detail & Related papers  (2022-01-06T18:57:59Z)
• Biphasic Face Photo-Sketch Synthesis via Semantic-Driven Generative Adversarial Network with Graph Representation Learning [40.544844623958426]
  We propose a novel Semantic-Driven Generative Adversarial Network to address the above issues. Considering that human faces have distinct spatial structures, we first inject class-wise semantic layouts into the generator. We construct two types of representational graphs via semantic parsing maps upon input faces, dubbed the IntrA-class Semantic Graph (IASG) and the InteR-class Structure Graph (IRSG)
  arXiv  Detail & Related papers  (2022-01-05T13:14:14Z)
• Unconstrained Face Sketch Synthesis via Perception-Adaptive Network and A New Benchmark [16.126100433405398]
  We argue that accurately perceiving facial region and facial components is crucial for unconstrained sketch synthesis. We propose a novel Perception-Adaptive Network (PANet), which can generate high-quality face sketches under unconstrained conditions. We introduce a new benchmark termed WildSketch, which contains 800 pairs of face photo-sketch with large variations in pose, expression, ethnic origin, background, and illumination.
  arXiv  Detail & Related papers  (2021-12-02T07:08:31Z)
• TANet: A new Paradigm for Global Face Super-resolution via Transformer-CNN Aggregation Network [72.41798177302175]
  We propose a novel paradigm based on the self-attention mechanism (i.e., the core of Transformer) to fully explore the representation capacity of the facial structure feature. Specifically, we design a Transformer-CNN aggregation network (TANet) consisting of two paths, in which one path uses CNNs responsible for restoring fine-grained facial details. By aggregating the features from the above two paths, the consistency of global facial structure and fidelity of local facial detail restoration are strengthened simultaneously.
  arXiv  Detail & Related papers  (2021-09-16T18:15:07Z)
• 3D Dense Geometry-Guided Facial Expression Synthesis by Adversarial Learning [54.24887282693925]
  We propose a novel framework to exploit 3D dense (depth and surface normals) information for expression manipulation. We use an off-the-shelf state-of-the-art 3D reconstruction model to estimate the depth and create a large-scale RGB-Depth dataset. Our experiments demonstrate that the proposed method outperforms the competitive baseline and existing arts by a large margin.
  arXiv  Detail & Related papers  (2020-09-30T17:12:35Z)
• InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANs [73.27299786083424]
  We propose a framework called InterFaceGAN to interpret the disentangled face representation learned by state-of-the-art GAN models. We first find that GANs learn various semantics in some linear subspaces of the latent space. We then conduct a detailed study on the correlation between different semantics and manage to better disentangle them via subspace projection.
  arXiv  Detail & Related papers  (2020-05-18T18:01:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.