Reinforced Disentanglement for Face Swapping without Skip Connection
- URL: http://arxiv.org/abs/2307.07928v4
- Date: Thu, 3 Aug 2023 06:05:02 GMT
- Title: Reinforced Disentanglement for Face Swapping without Skip Connection
- Authors: Xiaohang Ren, Xingyu Chen, Pengfei Yao, Heung-Yeung Shum, Baoyuan Wang
- Abstract summary: We introduce a new face swap framework called 'WSC-swap' that gets rid of skip connections and uses two target encoders.
Our results significantly outperform previous works on a rich set of metrics, including one novel metric for measuring identity consistency.
- Score: 18.97633893837313
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: The SOTA face swap models still suffer the problem of either target identity
(i.e., shape) being leaked or the target non-identity attributes (i.e.,
background, hair) failing to be fully preserved in the final results. We show
that this insufficient disentanglement is caused by two flawed designs that
were commonly adopted in prior models: (1) counting on only one compressed
encoder to represent both the semantic-level non-identity facial
attributes(i.e., pose) and the pixel-level non-facial region details, which is
contradictory to satisfy at the same time; (2) highly relying on long
skip-connections between the encoder and the final generator, leaking a certain
amount of target face identity into the result. To fix them, we introduce a new
face swap framework called 'WSC-swap' that gets rid of skip connections and
uses two target encoders to respectively capture the pixel-level non-facial
region attributes and the semantic non-identity attributes in the face region.
To further reinforce the disentanglement learning for the target encoder, we
employ both identity removal loss via adversarial training (i.e., GAN) and the
non-identity preservation loss via prior 3DMM models like [11]. Extensive
experiments on both FaceForensics++ and CelebA-HQ show that our results
significantly outperform previous works on a rich set of metrics, including one
novel metric for measuring identity consistency that was completely neglected
before.
Related papers
- 3D Face Modeling via Weakly-supervised Disentanglement Network joint Identity-consistency Prior [62.80458034704989]
Generative 3D face models featuring disentangled controlling factors hold immense potential for diverse applications in computer vision and computer graphics.
Previous 3D face modeling methods face a challenge as they demand specific labels to effectively disentangle these factors.
This paper introduces a Weakly-Supervised Disentanglement Framework, denoted as WSDF, to facilitate the training of controllable 3D face models without an overly stringent labeling requirement.
arXiv Detail & Related papers (2024-04-25T11:50:47Z) - Latent Diffusion Models for Attribute-Preserving Image Anonymization [4.080920304681247]
This paper presents the first approach to image anonymization based on Latent Diffusion Models (LDMs)
We propose two LDMs for this purpose: CAFLaGE-Base exploits a combination of pre-trained ControlNets, and a new controlling mechanism designed to increase the distance between the real and anonymized images.
arXiv Detail & Related papers (2024-03-21T19:09:21Z) - CLR-Face: Conditional Latent Refinement for Blind Face Restoration Using
Score-Based Diffusion Models [57.9771859175664]
Recent generative-prior-based methods have shown promising blind face restoration performance.
Generating fine-grained facial details faithful to inputs remains a challenging problem.
We introduce a diffusion-based-prior inside a VQGAN architecture that focuses on learning the distribution over uncorrupted latent embeddings.
arXiv Detail & Related papers (2024-02-08T23:51:49Z) - Attribute-preserving Face Dataset Anonymization via Latent Code
Optimization [64.4569739006591]
We present a task-agnostic anonymization procedure that directly optimize the images' latent representation in the latent space of a pre-trained GAN.
We demonstrate through a series of experiments that our method is capable of anonymizing the identity of the images whilst -- crucially -- better-preserving the facial attributes.
arXiv Detail & Related papers (2023-03-20T17:34:05Z) - Semantic-aware One-shot Face Re-enactment with Dense Correspondence
Estimation [100.60938767993088]
One-shot face re-enactment is a challenging task due to the identity mismatch between source and driving faces.
This paper proposes to use 3D Morphable Model (3DMM) for explicit facial semantic decomposition and identity disentanglement.
arXiv Detail & Related papers (2022-11-23T03:02:34Z) - Dual Spoof Disentanglement Generation for Face Anti-spoofing with Depth
Uncertainty Learning [54.15303628138665]
Face anti-spoofing (FAS) plays a vital role in preventing face recognition systems from presentation attacks.
Existing face anti-spoofing datasets lack diversity due to the insufficient identity and insignificant variance.
We propose Dual Spoof Disentanglement Generation framework to tackle this challenge by "anti-spoofing via generation"
arXiv Detail & Related papers (2021-12-01T15:36:59Z) - SimSwap: An Efficient Framework For High Fidelity Face Swapping [43.59969679039686]
We propose an efficient framework, called Simple Swap (SimSwap), aiming for generalized and high fidelity face swapping.
Our framework is capable of transferring the identity of an arbitrary source face into an arbitrary target face while preserving the attributes of the target face.
Experiments on wild faces demonstrate that our SimSwap is able to achieve competitive identity performance while preserving attributes better than previous state-of-the-art methods.
arXiv Detail & Related papers (2021-06-11T12:23:10Z) - Image Inpainting by End-to-End Cascaded Refinement with Mask Awareness [66.55719330810547]
Inpainting arbitrary missing regions is challenging because learning valid features for various masked regions is nontrivial.
We propose a novel mask-aware inpainting solution that learns multi-scale features for missing regions in the encoding phase.
Our framework is validated both quantitatively and qualitatively via extensive experiments on three public datasets.
arXiv Detail & Related papers (2021-04-28T13:17:47Z) - FaceShifter: Towards High Fidelity And Occlusion Aware Face Swapping [43.236261887752065]
We propose a novel two-stage framework, called FaceShifter, for high fidelity and occlusion aware face swapping.
In its first stage, our framework generates the swapped face in high-fidelity by exploiting and integrating the target attributes thoroughly and adaptively.
To address the challenging facial synthesiss, we append a second stage consisting of a novel Heuristic Error Acknowledging Refinement Network (HEAR-Net)
arXiv Detail & Related papers (2019-12-31T17:57:46Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.