Few-Shot Head Swapping in the Wild

• URL: http://arxiv.org/abs/2204.13100v1
• Date: Wed, 27 Apr 2022 17:52:51 GMT
• Title: Few-Shot Head Swapping in the Wild
• Authors: Changyong Shu, Hemao Wu, Hang Zhou, Jiaming Liu, Zhibin Hong, Changxing Ding, Junyu Han, Jingtuo Liu, Errui Ding, Jingdong Wang
• Abstract summary: The head swapping task aims at flawlessly placing a source head onto a target body, which is of great importance to various entertainment scenarios. It is inherently challenging due to its unique needs in head modeling and background blending. We present the Head Swapper (HeSer), which achieves few-shot head swapping in the wild through two delicately designed modules.
• Score: 79.78228139171574
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The head swapping task aims at flawlessly placing a source head onto a target body, which is of great importance to various entertainment scenarios. While face swapping has drawn much attention, the task of head swapping has rarely been explored, particularly under the few-shot setting. It is inherently challenging due to its unique needs in head modeling and background blending. In this paper, we present the Head Swapper (HeSer), which achieves few-shot head swapping in the wild through two delicately designed modules. Firstly, a Head2Head Aligner is devised to holistically migrate pose and expression information from the target to the source head by examining multi-scale information. Secondly, to tackle the challenges of skin color variations and head-background mismatches in the swapping procedure, a Head2Scene Blender is introduced to simultaneously modify facial skin color and fill mismatched gaps in the background around the head. Particularly, seamless blending is achieved with the help of a Semantic-Guided Color Reference Creation procedure and a Blending UNet. Extensive experiments demonstrate that the proposed method produces superior head swapping results in a variety of scenes.

Related papers

• GaussianHeads: End-to-End Learning of Drivable Gaussian Head Avatars from Coarse-to-fine Representations [54.94362657501809]
  We propose a new method to generate highly dynamic and deformable human head avatars from multi-view imagery in real-time. At the core of our method is a hierarchical representation of head models that allows to capture the complex dynamics of facial expressions and head movements. We train this coarse-to-fine facial avatar model along with the head pose as a learnable parameter in an end-to-end framework.
  arXiv  Detail & Related papers  (2024-09-18T13:05:43Z)
• HeadSculpt: Crafting 3D Head Avatars with Text [143.14548696613886]
  We introduce a versatile pipeline dubbed HeadSculpt for crafting 3D head avatars from textual prompts. We first equip the diffusion model with 3D awareness by leveraging landmark-based control and a learned textual embedding. We propose a novel identity-aware editing score distillation strategy to optimize a textured mesh with a high-resolution differentiable rendering technique.
  arXiv  Detail & Related papers  (2023-06-05T16:53:58Z)
• FONT: Flow-guided One-shot Talking Head Generation with Natural Head Motions [14.205344055665414]
  Flow-guided One-shot model achieves NaTural head motions over generated talking heads. Head pose prediction module is designed to generate head pose sequences from the source face and driving audio.
  arXiv  Detail & Related papers  (2023-03-31T03:25:06Z)
• DisCoHead: Audio-and-Video-Driven Talking Head Generation by Disentangled Control of Head Pose and Facial Expressions [21.064765388027727]
  DisCoHead is a novel method to disentangle and control head pose and facial expressions without supervision. DisCoHead successfully generates realistic audio-and-video-driven talking heads and outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-03-14T08:22:18Z)
• HS-Diffusion: Semantic-Mixing Diffusion for Head Swapping [150.06405071177048]
  We propose a semantic-mixing diffusion model for head swapping (HS-Diffusion) We blend the semantic layouts of source head and source body, and then inpaint the transition region by the semantic layout generator. We construct a new image-based head swapping benchmark and design two tailor-designed metrics.
  arXiv  Detail & Related papers  (2022-12-13T10:04:01Z)
• Learning to regulate 3D head shape by removing occluding hair from in-the-wild images [0.0]
  We present a novel approach for modeling the upper head by removing occluding hair and reconstructing the skin. Our unsupervised 3DMM model achieves state-of-the-art results on popular benchmarks.
  arXiv  Detail & Related papers  (2022-08-25T13:18:26Z)
• Head2HeadFS: Video-based Head Reenactment with Few-shot Learning [64.46913473391274]
  Head reenactment is a challenging task, which aims at transferring the entire head pose from a source person to a target. We propose head2headFS, a novel easily adaptable pipeline for head reenactment. Our video-based rendering network is fine-tuned under a few-shot learning strategy, using only a few samples.
  arXiv  Detail & Related papers  (2021-03-30T10:19:41Z)
• HeadGAN: One-shot Neural Head Synthesis and Editing [70.30831163311296]
  HeadGAN is a system that synthesises on 3D face representations and adapted to the facial geometry of any reference image. The 3D face representation enables HeadGAN to be further used as an efficient method for compression and reconstruction and a tool for expression and pose editing.
  arXiv  Detail & Related papers  (2020-12-15T12:51:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.