HS-Diffusion: Semantic-Mixing Diffusion for Head Swapping

• URL: http://arxiv.org/abs/2212.06458v3
• Date: Thu, 3 Aug 2023 07:32:30 GMT
• Title: HS-Diffusion: Semantic-Mixing Diffusion for Head Swapping
• Authors: Qinghe Wang, Lijie Liu, Miao Hua, Pengfei Zhu, Wangmeng Zuo, Qinghua Hu, Huchuan Lu, Bing Cao
• Abstract summary: We propose a semantic-mixing diffusion model for head swapping (HS-Diffusion) We blend the semantic layouts of source head and source body, and then inpaint the transition region by the semantic layout generator. We construct a new image-based head swapping benchmark and design two tailor-designed metrics.
• Score: 150.06405071177048
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Image-based head swapping task aims to stitch a source head to another source body flawlessly. This seldom-studied task faces two major challenges: 1) Preserving the head and body from various sources while generating a seamless transition region. 2) No paired head swapping dataset and benchmark so far. In this paper, we propose a semantic-mixing diffusion model for head swapping (HS-Diffusion) which consists of a latent diffusion model (LDM) and a semantic layout generator. We blend the semantic layouts of source head and source body, and then inpaint the transition region by the semantic layout generator, achieving a coarse-grained head swapping. Semantic-mixing LDM can further implement a fine-grained head swapping with the inpainted layout as condition by a progressive fusion process, while preserving head and body with high-quality reconstruction. To this end, we propose a semantic calibration strategy for natural inpainting and a neck alignment for geometric realism. Importantly, we construct a new image-based head swapping benchmark and design two tailor-designed metrics (Mask-FID and Focal-FID). Extensive experiments demonstrate the superiority of our framework. The code will be available: https://github.com/qinghew/HS-Diffusion.

Related papers

• Towards High-fidelity Head Blending with Chroma Keying for Industrial Applications [7.479901577089033]
  We introduce an industrial Head Blending pipeline for the task of seamlessly integrating an actor's head onto a target body in digital content creation. The key challenge stems from discrepancies in head shape and hair structure, which lead to unnatural boundaries and blending artifacts. We propose CHANGER, a novel pipeline that decouples background integration from foreground blending.
  arXiv  Detail & Related papers  (2024-11-01T15:14:59Z)
• Semi-Supervised Unconstrained Head Pose Estimation in the Wild [60.08319512840091]
  We propose the first semi-supervised unconstrained head pose estimation method SemiUHPE. Our method is based on the observation that the aspect-ratio invariant cropping of wild heads is superior to the previous landmark-based affine alignment. Experiments and ablation studies show that SemiUHPE outperforms existing methods greatly on public benchmarks.
  arXiv  Detail & Related papers  (2024-04-03T08:01:00Z)
• Face Swap via Diffusion Model [4.026688121914668]
  This report presents a diffusion model based framework for face swapping between two portrait images. The basic framework consists of three components, for face feature encoding, multi-conditional generation, and face inpainting respectively.
  arXiv  Detail & Related papers  (2024-03-02T07:02:17Z)
• HeadArtist: Text-conditioned 3D Head Generation with Self Score Distillation [95.58892028614444]
  This work presents HeadArtist for 3D head generation from text descriptions. We come up with an efficient pipeline that optimize a parameterized 3D head model under the supervision of the prior distillation. Experimental results suggest that our approach delivers high-quality 3D head sculptures with adequate geometry and photorealistic appearance.
  arXiv  Detail & Related papers  (2023-12-12T18:59:25Z)
• DPHMs: Diffusion Parametric Head Models for Depth-based Tracking [42.016598097736626]
  We introduce Diffusion Parametric Head Models (DPHMs) DPHMs are a generative model that enables robust volumetric head reconstruction and tracking from monocular depth sequences. We propose a latent diffusion-based prior to regularize volumetric head reconstruction and tracking.
  arXiv  Detail & Related papers  (2023-12-02T08:34:22Z)
• DisCoHead: Audio-and-Video-Driven Talking Head Generation by Disentangled Control of Head Pose and Facial Expressions [21.064765388027727]
  DisCoHead is a novel method to disentangle and control head pose and facial expressions without supervision. DisCoHead successfully generates realistic audio-and-video-driven talking heads and outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-03-14T08:22:18Z)
• Free-HeadGAN: Neural Talking Head Synthesis with Explicit Gaze Control [54.079327030892244]
  Free-HeadGAN is a person-generic neural talking head synthesis system. We show that modeling faces with sparse 3D facial landmarks are sufficient for achieving state-of-the-art generative performance.
  arXiv  Detail & Related papers  (2022-08-03T16:46:08Z)
• Dynamic Prototype Mask for Occluded Person Re-Identification [88.7782299372656]
  Existing methods mainly address this issue by employing body clues provided by an extra network to distinguish the visible part. We propose a novel Dynamic Prototype Mask (DPM) based on two self-evident prior knowledge. Under this condition, the occluded representation could be well aligned in a selected subspace spontaneously.
  arXiv  Detail & Related papers  (2022-07-19T03:31:13Z)
• Few-Shot Head Swapping in the Wild [79.78228139171574]
  The head swapping task aims at flawlessly placing a source head onto a target body, which is of great importance to various entertainment scenarios. It is inherently challenging due to its unique needs in head modeling and background blending. We present the Head Swapper (HeSer), which achieves few-shot head swapping in the wild through two delicately designed modules.
  arXiv  Detail & Related papers  (2022-04-27T17:52:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.