InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANs

• URL: http://arxiv.org/abs/2005.09635v2
• Date: Thu, 29 Oct 2020 08:36:47 GMT
• Title: InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANs
• Authors: Yujun Shen, Ceyuan Yang, Xiaoou Tang, Bolei Zhou
• Abstract summary: We propose a framework called InterFaceGAN to interpret the disentangled face representation learned by state-of-the-art GAN models. We first find that GANs learn various semantics in some linear subspaces of the latent space. We then conduct a detailed study on the correlation between different semantics and manage to better disentangle them via subspace projection.
• Score: 73.27299786083424
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Although Generative Adversarial Networks (GANs) have made significant progress in face synthesis, there lacks enough understanding of what GANs have learned in the latent representation to map a random code to a photo-realistic image. In this work, we propose a framework called InterFaceGAN to interpret the disentangled face representation learned by the state-of-the-art GAN models and study the properties of the facial semantics encoded in the latent space. We first find that GANs learn various semantics in some linear subspaces of the latent space. After identifying these subspaces, we can realistically manipulate the corresponding facial attributes without retraining the model. We then conduct a detailed study on the correlation between different semantics and manage to better disentangle them via subspace projection, resulting in more precise control of the attribute manipulation. Besides manipulating the gender, age, expression, and presence of eyeglasses, we can even alter the face pose and fix the artifacts accidentally made by GANs. Furthermore, we perform an in-depth face identity analysis and a layer-wise analysis to evaluate the editing results quantitatively. Finally, we apply our approach to real face editing by employing GAN inversion approaches and explicitly training feed-forward models based on the synthetic data established by InterFaceGAN. Extensive experimental results suggest that learning to synthesize faces spontaneously brings a disentangled and controllable face representation.

Related papers

• GaFET: Learning Geometry-aware Facial Expression Translation from In-The-Wild Images [55.431697263581626]
  We introduce a novel Geometry-aware Facial Expression Translation framework, which is based on parametric 3D facial representations and can stably decoupled expression. We achieve higher-quality and more accurate facial expression transfer results compared to state-of-the-art methods, and demonstrate applicability of various poses and complex textures.
  arXiv  Detail & Related papers  (2023-08-07T09:03:35Z)
• ExFaceGAN: Exploring Identity Directions in GAN's Learned Latent Space for Synthetic Identity Generation [16.494722503803196]
  We propose a framework, ExFaceGAN, to disentangle identity information in pretrained GANs latent spaces. By sampling from each side of the boundary, our ExFaceGAN can generate multiple samples of synthetic identity. As an example, we empirically prove that data generated by ExFaceGAN can be successfully used to train face recognition models.
  arXiv  Detail & Related papers  (2023-07-11T10:14:41Z)
• Spatial Steerability of GANs via Self-Supervision from Discriminator [123.27117057804732]
  We propose a self-supervised approach to improve the spatial steerability of GANs without searching for steerable directions in the latent space. Specifically, we design randomly sampled Gaussian heatmaps to be encoded into the intermediate layers of generative models as spatial inductive bias. During inference, users can interact with the spatial heatmaps in an intuitive manner, enabling them to edit the output image by adjusting the scene layout, moving, or removing objects.
  arXiv  Detail & Related papers  (2023-01-20T07:36:29Z)
• Discovering Class-Specific GAN Controls for Semantic Image Synthesis [73.91655061467988]
  We propose a novel method for finding spatially disentangled class-specific directions in the latent space of pretrained SIS models. We show that the latent directions found by our method can effectively control the local appearance of semantic classes.
  arXiv  Detail & Related papers  (2022-12-02T21:39:26Z)
• Detecting High-Quality GAN-Generated Face Images using Neural Networks [23.388645531702597]
  We propose a new strategy to differentiate GAN-generated images from authentic images by leveraging spectral band discrepancies. In particular, we enable the digital preservation of face images using the Cross-band co-occurrence matrix and spatial co-occurrence matrix. We show that the performance boost is particularly significant and achieves more than 92% in different post-processing environments.
  arXiv  Detail & Related papers  (2022-03-03T13:53:27Z)
• Synthesizing Human Faces using Latent Space Factorization and Local Weights (Extended Version) [24.888957468547744]
  The proposed model allows partial manipulation of the face while still learning the whole face mesh. We factorize the latent space of the whole face to the subspace indicating different parts of the face.
  arXiv  Detail & Related papers  (2021-07-19T10:17:30Z)
• Heterogeneous Face Frontalization via Domain Agnostic Learning [74.86585699909459]
  We propose a domain agnostic learning-based generative adversarial network (DAL-GAN) which can synthesize frontal views in the visible domain from thermal faces with pose variations. DAL-GAN consists of a generator with an auxiliary classifier and two discriminators which capture both local and global texture discriminations for better synthesis.
  arXiv  Detail & Related papers  (2021-07-17T20:41:41Z)
• LARNet: Lie Algebra Residual Network for Profile Face Recognition [5.968418413932049]
  We propose a novel method with Lie algebra theory to explore how face rotation in the 3D space affects the deep feature generation process of convolutional neural networks (CNNs) We prove that face rotation in the image space is equivalent to an additive residual component in the feature space of CNNs, which is determined solely by the rotation. Our LARNet design consists of a residual for decoding rotation information from input face images, and a gating magnitude to learn rotation for controlling the number of residual components contributing to the feature learning process.
  arXiv  Detail & Related papers  (2021-03-15T05:44:54Z)
• Disentangled and Controllable Face Image Generation via 3D Imitative-Contrastive Learning [43.53235319568048]
  We propose DiscoFaceGAN, an approach for face image generation of virtual people with disentangled, precisely-controllable latent representations. We embed 3D priors into adversarial learning and train the network to imitate the image formation of an analytic 3D face deformation and rendering process.
  arXiv  Detail & Related papers  (2020-04-24T11:21:13Z)
• DotFAN: A Domain-transferred Face Augmentation Network for Pose and Illumination Invariant Face Recognition [94.96686189033869]
  We propose a 3D model-assisted domain-transferred face augmentation network (DotFAN) DotFAN can generate a series of variants of an input face based on the knowledge distilled from existing rich face datasets collected from other domains. Experiments show that DotFAN is beneficial for augmenting small face datasets to improve their within-class diversity.
  arXiv  Detail & Related papers  (2020-02-23T08:16:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.