Unsupervised High-Resolution Portrait Gaze Correction and Animation

• URL: http://arxiv.org/abs/2207.00256v1
• Date: Fri, 1 Jul 2022 08:14:42 GMT
• Title: Unsupervised High-Resolution Portrait Gaze Correction and Animation
• Authors: Jichao Zhang, Jingjing Chen, Hao Tang, Enver Sangineto, Peng Wu, Yan Yan, Nicu Sebe, Wei Wang
• Abstract summary: This paper proposes a gaze correction and animation method for high-resolution, unconstrained portrait images. We first create two new portrait datasets: CelebGaze and high-resolution CelebHQGaze. We formulate the gaze correction task as an image inpainting problem, addressed using a Gaze Correction Module and a Gaze Animation Module.
• Score: 81.19271523855554
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper proposes a gaze correction and animation method for high-resolution, unconstrained portrait images, which can be trained without the gaze angle and the head pose annotations. Common gaze-correction methods usually require annotating training data with precise gaze, and head pose information. Solving this problem using an unsupervised method remains an open problem, especially for high-resolution face images in the wild, which are not easy to annotate with gaze and head pose labels. To address this issue, we first create two new portrait datasets: CelebGaze and high-resolution CelebHQGaze. Second, we formulate the gaze correction task as an image inpainting problem, addressed using a Gaze Correction Module (GCM) and a Gaze Animation Module (GAM). Moreover, we propose an unsupervised training strategy, i.e., Synthesis-As-Training, to learn the correlation between the eye region features and the gaze angle. As a result, we can use the learned latent space for gaze animation with semantic interpolation in this space. Moreover, to alleviate both the memory and the computational costs in the training and the inference stage, we propose a Coarse-to-Fine Module (CFM) integrated with GCM and GAM. Extensive experiments validate the effectiveness of our method for both the gaze correction and the gaze animation tasks in both low and high-resolution face datasets in the wild and demonstrate the superiority of our method with respect to the state of the arts. Code is available at https://github.com/zhangqianhui/GazeAnimationV2

Related papers

• Toward Super-Resolution for Appearance-Based Gaze Estimation [4.594159253008448]
  Super-resolution has been shown to improve image quality from a visual perspective. We propose a two-step framework based on SwinIR super-resolution model. Self-supervised learning aims to learn from unlabelled data to reduce the amount of required labeled data for downstream tasks.
  arXiv  Detail & Related papers  (2023-03-17T17:40:32Z)
• Joint Learning of Portrait Intrinsic Decomposition and Relighting [5.601217969637838]
  Inverse rendering is the problem of decomposing an image into its intrinsic components, i.e. albedo, normal and lighting. Here, we propose a new self-supervised training paradigm that reduces the need for full supervision on the decomposition task. We show-case the effectiveness of our training paradigm on both intrinsic decomposition and relighting and demonstrate how the model struggles in both tasks without the self-supervised loss terms.
  arXiv  Detail & Related papers  (2021-06-22T19:40:22Z)
• Bridging Composite and Real: Towards End-to-end Deep Image Matting [88.79857806542006]
  We study the roles of semantics and details for image matting. We propose a novel Glance and Focus Matting network (GFM), which employs a shared encoder and two separate decoders. Comprehensive empirical studies have demonstrated that GFM outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-10-30T10:57:13Z)
• Self-Learning Transformations for Improving Gaze and Head Redirection [49.61091281780071]
  We propose a novel generative model for images of faces, that is capable of producing high-quality images under fine-grained control over eye gaze and head orientation angles. This requires the disentangling of many appearance related factors including gaze and head orientation but also lighting, hue etc. We show that explicitly disentangling task-irrelevant factors results in more accurate modelling of gaze and head orientation.
  arXiv  Detail & Related papers  (2020-10-23T11:18:37Z)
• Dual In-painting Model for Unsupervised Gaze Correction and Animation in the Wild [82.42401132933462]
  We present a solution that works without the need for precise annotations of the gaze angle and the head pose. Our method consists of three novel modules: the Gaze Correction module (GCM), the Gaze Animation module (GAM), and the Pretrained Autoencoder module (PAM)
  arXiv  Detail & Related papers  (2020-08-09T23:14:16Z)
• Perspective Plane Program Induction from a Single Image [85.28956922100305]
  We study the inverse graphics problem of inferring a holistic representation for natural images. We formulate this problem as jointly finding the camera pose and scene structure that best describe the input image. Our proposed framework, Perspective Plane Program Induction (P3I), combines search-based and gradient-based algorithms to efficiently solve the problem.
  arXiv  Detail & Related papers  (2020-06-25T21:18:58Z)
• Coarse-to-Fine Gaze Redirection with Numerical and Pictorial Guidance [74.27389895574422]
  We propose a novel gaze redirection framework which exploits both a numerical and a pictorial direction guidance. The proposed method outperforms the state-of-the-art approaches in terms of both image quality and redirection precision.
  arXiv  Detail & Related papers  (2020-04-07T01:17:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.