Fine-Grained Expression Manipulation via Structured Latent Space

• URL: http://arxiv.org/abs/2004.09769v2
• Date: Sun, 10 May 2020 08:11:12 GMT
• Title: Fine-Grained Expression Manipulation via Structured Latent Space
• Authors: Junshu Tang, Zhiwen Shao, Lizhuang Ma
• Abstract summary: We propose an end-to-end expression-guided generative adversarial network (EGGAN) to manipulate fine-grained expressions. Our method can manipulate fine-grained expressions, and generate continuous intermediate expressions between source and target expressions.
• Score: 30.789513209376032
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Fine-grained facial expression manipulation is a challenging problem, as fine-grained expression details are difficult to be captured. Most existing expression manipulation methods resort to discrete expression labels, which mainly edit global expressions and ignore the manipulation of fine details. To tackle this limitation, we propose an end-to-end expression-guided generative adversarial network (EGGAN), which utilizes structured latent codes and continuous expression labels as input to generate images with expected expressions. Specifically, we adopt an adversarial autoencoder to map a source image into a structured latent space. Then, given the source latent code and the target expression label, we employ a conditional GAN to generate a new image with the target expression. Moreover, we introduce a perceptual loss and a multi-scale structural similarity loss to preserve identity and global shape during generation. Extensive experiments show that our method can manipulate fine-grained expressions, and generate continuous intermediate expressions between source and target expressions.

Related papers

• LatentAvatar: Learning Latent Expression Code for Expressive Neural Head Avatar [60.363572621347565]
  We present LatentAvatar, an expressive neural head avatar driven by latent expression codes. LatentAvatar is able to capture challenging expressions and the subtle movement of teeth and even eyeballs.
  arXiv  Detail & Related papers  (2023-05-02T03:49:12Z)
• Weakly-supervised segmentation of referring expressions [81.73850439141374]
  Text grounded semantic SEGmentation learns segmentation masks directly from image-level referring expressions without pixel-level annotations. Our approach demonstrates promising results for weakly-supervised referring expression segmentation on the PhraseCut and RefCOCO datasets.
  arXiv  Detail & Related papers  (2022-05-10T07:52:24Z)
• IR-GAN: Image Manipulation with Linguistic Instruction by Increment Reasoning [110.7118381246156]
  Increment Reasoning Generative Adversarial Network (IR-GAN) aims to reason consistency between visual increment in images and semantic increment in instructions. First, we introduce the word-level and instruction-level instruction encoders to learn user's intention from history-correlated instructions as semantic increment. Second, we embed the representation of semantic increment into that of source image for generating target image, where source image plays the role of referring auxiliary.
  arXiv  Detail & Related papers  (2022-04-02T07:48:39Z)
• StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery [71.1862388442953]
  We develop a text-based interface for StyleGAN image manipulation. We first introduce an optimization scheme that utilizes a CLIP-based loss to modify an input latent vector in response to a user-provided text prompt. Next, we describe a latent mapper that infers a text-guided latent manipulation step for a given input image, allowing faster and more stable text-based manipulation.
  arXiv  Detail & Related papers  (2021-03-31T17:51:25Z)
• Identity-aware Facial Expression Recognition in Compressed Video [27.14473209125735]
  In the up to two orders of magnitude compressed domain, we can explicitly infer the expression from the residual frames. We do not need the identity label or multiple expression samples from the same person for identity elimination. Our solution can achieve comparable or better performance than the recent decoded image based methods.
  arXiv  Detail & Related papers  (2021-01-01T21:03:13Z)
• Mutual Information Regularized Identity-aware Facial ExpressionRecognition in Compressed Video [27.602648102881535]
  We propose a novel collaborative min-min game for mutual information (MI) minimization in latent space. We do not need the identity label or multiple expression samples from the same person for identity elimination. Our solution can achieve comparable or better performance than the recent decoded image-based methods.
  arXiv  Detail & Related papers  (2020-10-20T21:42:18Z)
• LEED: Label-Free Expression Editing via Disentanglement [57.09545215087179]
  LEED framework is capable of editing the expression of both frontal and profile facial images without requiring any expression label. Two novel losses are designed for optimal expression disentanglement and consistent synthesis.
  arXiv  Detail & Related papers  (2020-07-17T13:36:15Z)
• An Efficient Integration of Disentangled Attended Expression and Identity FeaturesFor Facial Expression Transfer andSynthesis [6.383596973102899]
  We present an Attention-based Identity Preserving Generative Adversarial Network (AIP-GAN) to overcome the identity leakage problem from a source image to a generated face image. Our key insight is that the identity preserving network should be able to disentangle and compose shape, appearance, and expression information for efficient facial expression transfer and synthesis.
  arXiv  Detail & Related papers  (2020-05-01T17:14:53Z)
• Cascade EF-GAN: Progressive Facial Expression Editing with Local Focuses [49.077232276128754]
  We propose a novel network that performs progressive facial expression editing with local expression focuses. The introduction of the local focus enables the Cascade EF-GAN to better preserve identity-related features. In addition, an innovative cascade transformation strategy is designed by dividing a large facial expression transformation into multiple small ones in cascade.
  arXiv  Detail & Related papers  (2020-03-12T17:11:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.