LatentSwap: An Efficient Latent Code Mapping Framework for Face Swapping
- URL: http://arxiv.org/abs/2402.18351v2
- Date: Sat, 2 Mar 2024 14:08:03 GMT
- Title: LatentSwap: An Efficient Latent Code Mapping Framework for Face Swapping
- Authors: Changho Choi, Minho Kim, Junhyeok Lee, Hyoung-Kyu Song, Younggeun Kim,
Seungryong Kim
- Abstract summary: We propose LatentSwap, a framework generating a face swap latent code of a given generator.
Our framework is light and does not require datasets besides employing the pre-trained models.
We show that our framework is applicable to other generators such as StyleNeRF, paving a way to 3D-aware face swapping.
- Score: 41.59473130977597
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: We propose LatentSwap, a simple face swapping framework generating a face
swap latent code of a given generator. Utilizing randomly sampled latent codes,
our framework is light and does not require datasets besides employing the
pre-trained models, with the training procedure also being fast and
straightforward. The loss objective consists of only three terms, and can
effectively control the face swap results between source and target images. By
attaching a pre-trained GAN inversion model independent to the model and using
the StyleGAN2 generator, our model produces photorealistic and high-resolution
images comparable to other competitive face swap models. We show that our
framework is applicable to other generators such as StyleNeRF, paving a way to
3D-aware face swapping and is also compatible with other downstream StyleGAN2
generator tasks. The source code and models can be found at
\url{https://github.com/usingcolor/LatentSwap}.
Related papers
- Face Swap via Diffusion Model [4.026688121914668]
This report presents a diffusion model based framework for face swapping between two portrait images.
The basic framework consists of three components, for face feature encoding, multi-conditional generation, and face inpainting respectively.
arXiv Detail & Related papers (2024-03-02T07:02:17Z) - Controllable 3D Face Generation with Conditional Style Code Diffusion [51.24656496304069]
TEx-Face(TExt & Expression-to-Face) addresses challenges by dividing the task into three components, i.e., 3D GAN Inversion, Conditional Style Code Diffusion, and 3D Face Decoding.
Experiments conducted on FFHQ, CelebA-HQ, and CelebA-Dialog demonstrate the promising performance of our TEx-Face.
arXiv Detail & Related papers (2023-12-21T15:32:49Z) - Controlling the Output of a Generative Model by Latent Feature Vector
Shifting [0.0]
We present our novel method for latent vector shifting for controlled output image modification.
In our approach we use a pre-trained model of StyleGAN3 that generates images of realistic human faces.
Our latent feature shifter is a neural network model with a task to shift the latent vectors of a generative model into a specified feature direction.
arXiv Detail & Related papers (2023-11-15T10:42:06Z) - DCFace: Synthetic Face Generation with Dual Condition Diffusion Model [18.662943303044315]
We propose a Dual Condition Face Generator (DCFace) based on a diffusion model.
Our novel Patch-wise style extractor and Time-step dependent ID loss enables DCFace to consistently produce face images of the same subject under different styles with precise control.
arXiv Detail & Related papers (2023-04-14T11:31:49Z) - CGOF++: Controllable 3D Face Synthesis with Conditional Generative
Occupancy Fields [52.14985242487535]
We propose a new conditional 3D face synthesis framework, which enables 3D controllability over generated face images.
At its core is a conditional Generative Occupancy Field (cGOF++) that effectively enforces the shape of the generated face to conform to a given 3D Morphable Model (3DMM) mesh.
Experiments validate the effectiveness of the proposed method and show more precise 3D controllability than state-of-the-art 2D-based controllable face synthesis methods.
arXiv Detail & Related papers (2022-11-23T19:02:50Z) - StyleSwap: Style-Based Generator Empowers Robust Face Swapping [90.05775519962303]
We introduce a concise and effective framework named StyleSwap.
Our core idea is to leverage a style-based generator to empower high-fidelity and robust face swapping.
We identify that with only minimal modifications, a StyleGAN2 architecture can successfully handle the desired information from both source and target.
arXiv Detail & Related papers (2022-09-27T16:35:16Z) - FairStyle: Debiasing StyleGAN2 with Style Channel Manipulations [0.0]
We propose a method for modifying a pre-trained StyleGAN2 model that can be used to generate a balanced set of images.
Our method successfully debiases the GAN model within a few minutes without compromising the quality of the generated images.
arXiv Detail & Related papers (2022-02-13T07:39:48Z) - Controllable and Compositional Generation with Latent-Space Energy-Based
Models [60.87740144816278]
Controllable generation is one of the key requirements for successful adoption of deep generative models in real-world applications.
In this work, we use energy-based models (EBMs) to handle compositional generation over a set of attributes.
By composing energy functions with logical operators, this work is the first to achieve such compositionality in generating photo-realistic images of resolution 1024x1024.
arXiv Detail & Related papers (2021-10-21T03:31:45Z) - Lifting 2D StyleGAN for 3D-Aware Face Generation [52.8152883980813]
We propose a framework, called LiftedGAN, that disentangles and lifts a pre-trained StyleGAN2 for 3D-aware face generation.
Our model is "3D-aware" in the sense that it is able to (1) disentangle the latent space of StyleGAN2 into texture, shape, viewpoint, lighting and (2) generate 3D components for synthetic images.
arXiv Detail & Related papers (2020-11-26T05:02:09Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.