Everything's Talkin': Pareidolia Face Reenactment
- URL: http://arxiv.org/abs/2104.03061v1
- Date: Wed, 7 Apr 2021 11:19:13 GMT
- Title: Everything's Talkin': Pareidolia Face Reenactment
- Authors: Linsen Song, Wayne Wu, Chaoyou Fu, Chen Qian, Chen Change Loy, Ran He
- Abstract summary: Pareidolia Face Reenactment is defined as animating a static illusory face to move in tandem with a human face in the video.
For the large differences between pareidolia face reenactment and traditional human face reenactment, shape variance and texture variance are introduced.
We propose a novel Parametric Unsupervised Reenactment Algorithm to tackle these two challenges.
- Score: 119.49707201178633
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We present a new application direction named Pareidolia Face Reenactment,
which is defined as animating a static illusory face to move in tandem with a
human face in the video. For the large differences between pareidolia face
reenactment and traditional human face reenactment, two main challenges are
introduced, i.e., shape variance and texture variance. In this work, we propose
a novel Parametric Unsupervised Reenactment Algorithm to tackle these two
challenges. Specifically, we propose to decompose the reenactment into three
catenate processes: shape modeling, motion transfer and texture synthesis. With
the decomposition, we introduce three crucial components, i.e., Parametric
Shape Modeling, Expansionary Motion Transfer and Unsupervised Texture
Synthesizer, to overcome the problems brought by the remarkably variances on
pareidolia faces. Extensive experiments show the superior performance of our
method both qualitatively and quantitatively. Code, model and data are
available on our project page.
Related papers
- GaussianHeads: End-to-End Learning of Drivable Gaussian Head Avatars from Coarse-to-fine Representations [54.94362657501809]
We propose a new method to generate highly dynamic and deformable human head avatars from multi-view imagery in real-time.
At the core of our method is a hierarchical representation of head models that allows to capture the complex dynamics of facial expressions and head movements.
We train this coarse-to-fine facial avatar model along with the head pose as a learnable parameter in an end-to-end framework.
arXiv Detail & Related papers (2024-09-18T13:05:43Z) - HyperReenact: One-Shot Reenactment via Jointly Learning to Refine and
Retarget Faces [47.27033282706179]
We present our method for neural face reenactment, called HyperReenact, that aims to generate realistic talking head images of a source identity.
Our method operates under the one-shot setting (i.e., using a single source frame) and allows for cross-subject reenactment, without requiring subject-specific fine-tuning.
We compare our method both quantitatively and qualitatively against several state-of-the-art techniques on the standard benchmarks of VoxCeleb1 and VoxCeleb2.
arXiv Detail & Related papers (2023-07-20T11:59:42Z) - Parametric Implicit Face Representation for Audio-Driven Facial
Reenactment [52.33618333954383]
We propose a novel audio-driven facial reenactment framework that is both controllable and can generate high-quality talking heads.
Specifically, our parametric implicit representation parameterizes the implicit representation with interpretable parameters of 3D face models.
Our method can generate more realistic results than previous methods with greater fidelity to the identities and talking styles of speakers.
arXiv Detail & Related papers (2023-06-13T07:08:22Z) - Semantic-aware One-shot Face Re-enactment with Dense Correspondence
Estimation [100.60938767993088]
One-shot face re-enactment is a challenging task due to the identity mismatch between source and driving faces.
This paper proposes to use 3D Morphable Model (3DMM) for explicit facial semantic decomposition and identity disentanglement.
arXiv Detail & Related papers (2022-11-23T03:02:34Z) - FSGANv2: Improved Subject Agnostic Face Swapping and Reenactment [28.83743270895698]
We present Face Swapping GAN (FSGAN) for face swapping and reenactment.
Unlike previous work, we offer a subject swapping scheme that can be applied to pairs of faces without requiring training on those faces.
We derive a novel iterative deep learning--based approach for face reenactment which adjusts significant pose and expression variations that can be applied to a single image or a video sequence.
For video sequences, we introduce a continuous agnostic of the face views based on reenactment, Delaunay Triangulation, and bary coordinates. Occluded face regions are handled by a face completion network.
arXiv Detail & Related papers (2022-02-25T21:04:39Z) - Thinking the Fusion Strategy of Multi-reference Face Reenactment [4.1509697008011175]
We show that simple extension by using multiple reference images significantly improves generation quality.
We show this by 1) conducting the reconstruction task on publicly available dataset, 2) conducting facial motion transfer on our original dataset which consists of multi-person's head movement video sequences, and 3) using a newly proposed evaluation metric to validate that our method achieves better quantitative results.
arXiv Detail & Related papers (2022-02-22T09:17:26Z) - Wish You Were Here: Context-Aware Human Generation [100.51309746913512]
We present a novel method for inserting objects, specifically humans, into existing images.
Our method involves threeworks: the first generates the semantic map of the new person, given the pose of the other persons in the scene.
The second network renders the pixels of the novel person and its blending mask, based on specifications in the form of multiple appearance components.
A third network refines the generated face in order to match those of the target person.
arXiv Detail & Related papers (2020-05-21T14:09:14Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.