Coarse-to-Fine Gaze Redirection with Numerical and Pictorial Guidance

• URL: http://arxiv.org/abs/2004.03064v4
• Date: Thu, 26 Nov 2020 06:17:15 GMT
• Title: Coarse-to-Fine Gaze Redirection with Numerical and Pictorial Guidance
• Authors: Jingjing Chen, Jichao Zhang, Enver Sangineto, Jiayuan Fan, Tao Chen, Nicu Sebe
• Abstract summary: We propose a novel gaze redirection framework which exploits both a numerical and a pictorial direction guidance. The proposed method outperforms the state-of-the-art approaches in terms of both image quality and redirection precision.
• Score: 74.27389895574422
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Gaze redirection aims at manipulating the gaze of a given face image with respect to a desired direction (i.e., a reference angle) and it can be applied to many real life scenarios, such as video-conferencing or taking group photos. However, previous work on this topic mainly suffers of two limitations: (1) Low-quality image generation and (2) Low redirection precision. In this paper, we propose to alleviate these problems by means of a novel gaze redirection framework which exploits both a numerical and a pictorial direction guidance, jointly with a coarse-to-fine learning strategy. Specifically, the coarse branch learns the spatial transformation which warps input image according to desired gaze. On the other hand, the fine-grained branch consists of a generator network with conditional residual image learning and a multi-task discriminator. This second branch reduces the gap between the previously warped image and the ground-truth image and recovers finer texture details. Moreover, we propose a numerical and pictorial guidance module~(NPG) which uses a pictorial gazemap description and numerical angles as an extra guide to further improve the precision of gaze redirection. Extensive experiments on a benchmark dataset show that the proposed method outperforms the state-of-the-art approaches in terms of both image quality and redirection precision. The code is available at https://github.com/jingjingchen777/CFGR

Related papers

• Image-GS: Content-Adaptive Image Representation via 2D Gaussians [55.15950594752051]
  We propose Image-GS, a content-adaptive image representation. Using anisotropic 2D Gaussians as the basis, Image-GS shows high memory efficiency, supports fast random access, and offers a natural level of detail stack. General efficiency and fidelity of Image-GS are validated against several recent neural image representations and industry-standard texture compressors. We hope this research offers insights for developing new applications that require adaptive quality and resource control, such as machine perception, asset streaming, and content generation.
  arXiv  Detail & Related papers  (2024-07-02T00:45:21Z)
• UCDFormer: Unsupervised Change Detection Using a Transformer-driven Image Translation [20.131754484570454]
  Change detection (CD) by comparing two bi-temporal images is a crucial task in remote sensing. We propose a change detection with domain shift setting for remote sensing images. We present a novel unsupervised CD method using a light-weight transformer, called UCDFormer.
  arXiv  Detail & Related papers  (2023-08-02T13:39:08Z)
• Diffusion-Based Scene Graph to Image Generation with Masked Contrastive Pre-Training [112.94542676251133]
  We propose to learn scene graph embeddings by directly optimizing their alignment with images. Specifically, we pre-train an encoder to extract both global and local information from scene graphs. The resulting method, called SGDiff, allows for the semantic manipulation of generated images by modifying scene graph nodes and connections.
  arXiv  Detail & Related papers  (2022-11-21T01:11:19Z)
• CUDA-GR: Controllable Unsupervised Domain Adaptation for Gaze Redirection [3.0141238193080295]
  The aim of gaze redirection is to manipulate the gaze in an image to the desired direction. Advancement in generative adversarial networks has shown excellent results in generating photo-realistic images. To enable such fine-tuned control, one needs to obtain ground truth annotations for the training data which can be very expensive.
  arXiv  Detail & Related papers  (2021-06-21T04:39:42Z)
• Self-Learning Transformations for Improving Gaze and Head Redirection [49.61091281780071]
  We propose a novel generative model for images of faces, that is capable of producing high-quality images under fine-grained control over eye gaze and head orientation angles. This requires the disentangling of many appearance related factors including gaze and head orientation but also lighting, hue etc. We show that explicitly disentangling task-irrelevant factors results in more accurate modelling of gaze and head orientation.
  arXiv  Detail & Related papers  (2020-10-23T11:18:37Z)
• Controllable Continuous Gaze Redirection [47.15883248953411]
  We present interpGaze, a novel framework for controllable gaze redirection. Our goal is to redirect the eye gaze of one person into any gaze direction depicted in the reference image. The proposed interpGaze outperforms state-of-the-art methods in terms of image quality and redirection precision.
  arXiv  Detail & Related papers  (2020-10-09T11:50:06Z)
• RANSAC-Flow: generic two-stage image alignment [53.11926395028508]
  We show that a simple unsupervised approach performs surprisingly well across a range of tasks. Despite its simplicity, our method shows competitive results on a range of tasks and datasets.
  arXiv  Detail & Related papers  (2020-04-03T12:37:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.