Cross-Domain and Disentangled Face Manipulation with 3D Guidance
- URL: http://arxiv.org/abs/2104.11228v1
- Date: Thu, 22 Apr 2021 17:59:50 GMT
- Title: Cross-Domain and Disentangled Face Manipulation with 3D Guidance
- Authors: Can Wang and Menglei Chai and Mingming He and Dongdong Chen and Jing
Liao
- Abstract summary: We propose the first method to manipulate faces in arbitrary domains using human 3DMM.
This is achieved through two major steps: 1) disentangled mapping from 3DMM parameters to the latent space embedding of a pre-trained StyleGAN2.
Experiments and comparisons demonstrate the superiority of our high-quality semantic manipulation method on a variety of face domains.
- Score: 33.43993665841577
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Face image manipulation via three-dimensional guidance has been widely
applied in various interactive scenarios due to its semantically-meaningful
understanding and user-friendly controllability. However, existing
3D-morphable-model-based manipulation methods are not directly applicable to
out-of-domain faces, such as non-photorealistic paintings, cartoon portraits,
or even animals, mainly due to the formidable difficulties in building the
model for each specific face domain. To overcome this challenge, we propose, as
far as we know, the first method to manipulate faces in arbitrary domains using
human 3DMM. This is achieved through two major steps: 1) disentangled mapping
from 3DMM parameters to the latent space embedding of a pre-trained StyleGAN2
that guarantees disentangled and precise controls for each semantic attribute;
and 2) cross-domain adaptation that bridges domain discrepancies and makes
human 3DMM applicable to out-of-domain faces by enforcing a consistent latent
space embedding. Experiments and comparisons demonstrate the superiority of our
high-quality semantic manipulation method on a variety of face domains with all
major 3D facial attributes controllable: pose, expression, shape, albedo, and
illumination. Moreover, we develop an intuitive editing interface to support
user-friendly control and instant feedback. Our project page is
https://cassiepython.github.io/sigasia/cddfm3d.html.
Related papers
- Revealing Directions for Text-guided 3D Face Editing [52.85632020601518]
3D face editing is a significant task in multimedia, aimed at the manipulation of 3D face models across various control signals.
We present Face Clan, a text-general approach for generating and manipulating 3D faces based on arbitrary attribute descriptions.
Our method offers a precisely controllable manipulation method, allowing users to intuitively customize regions of interest with the text description.
arXiv Detail & Related papers (2024-10-07T12:04:39Z) - Single Image, Any Face: Generalisable 3D Face Generation [59.9369171926757]
We propose a novel model, Gen3D-Face, which generates 3D human faces with unconstrained single image input.
To the best of our knowledge, this is the first attempt and benchmark for creating photorealistic 3D human face avatars from single images.
arXiv Detail & Related papers (2024-09-25T14:56:37Z) - Decaf: Monocular Deformation Capture for Face and Hand Interactions [77.75726740605748]
This paper introduces the first method that allows tracking human hands interacting with human faces in 3D from single monocular RGB videos.
We model hands as articulated objects inducing non-rigid face deformations during an active interaction.
Our method relies on a new hand-face motion and interaction capture dataset with realistic face deformations acquired with a markerless multi-view camera system.
arXiv Detail & Related papers (2023-09-28T17:59:51Z) - Single-Shot Implicit Morphable Faces with Consistent Texture
Parameterization [91.52882218901627]
We propose a novel method for constructing implicit 3D morphable face models that are both generalizable and intuitive for editing.
Our method improves upon photo-realism, geometry, and expression accuracy compared to state-of-the-art methods.
arXiv Detail & Related papers (2023-05-04T17:58:40Z) - Controllable Face Manipulation and UV Map Generation by Self-supervised
Learning [20.10160338724354]
Recent methods achieve explicit control over 2D images by combining 2D generative model and 3DMM.
Due to the lack of realism and clarity in texture reconstruction by 3DMM, there is a domain gap between the synthetic image and the rendered image of 3DMM.
In this study, we propose to explicitly edit the latent space of the pretrained StyleGAN by controlling the parameters of the 3DMM.
arXiv Detail & Related papers (2022-09-24T16:49:25Z) - 3D-FM GAN: Towards 3D-Controllable Face Manipulation [43.99393180444706]
3D-FM GAN is a novel conditional GAN framework designed specifically for 3D-controllable face manipulation.
By carefully encoding both the input face image and a physically-based rendering of 3D edits into a StyleGAN's latent spaces, our image generator provides high-quality, identity-preserved, 3D-controllable face manipulation.
We show that our method outperforms the prior arts on various tasks, with better editability, stronger identity preservation, and higher photo-realism.
arXiv Detail & Related papers (2022-08-24T01:33:13Z) - MOST-GAN: 3D Morphable StyleGAN for Disentangled Face Image Manipulation [69.35523133292389]
We propose a framework that a priori models physical attributes of the face explicitly, thus providing disentanglement by design.
Our method, MOST-GAN, integrates the expressive power and photorealism of style-based GANs with the physical disentanglement and flexibility of nonlinear 3D morphable models.
It achieves photorealistic manipulation of portrait images with fully disentangled 3D control over their physical attributes, enabling extreme manipulation of lighting, facial expression, and pose variations up to full profile view.
arXiv Detail & Related papers (2021-11-01T15:53:36Z) - StyleRig: Rigging StyleGAN for 3D Control over Portrait Images [81.43265493604302]
StyleGAN generates portrait images of faces with eyes, teeth, hair and context (neck, shoulders, background)
StyleGAN lacks a rig-like control over semantic face parameters that are interpretable in 3D, such as face pose, expressions, and scene illumination.
We present the first method to provide a face rig-like control over a pretrained and fixed StyleGAN via a 3DMM.
arXiv Detail & Related papers (2020-03-31T21:20:34Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.