What to Preserve and What to Transfer: Faithful, Identity-Preserving Diffusion-based Hairstyle Transfer

• URL: http://arxiv.org/abs/2408.16450v1
• Date: Thu, 29 Aug 2024 11:30:21 GMT
• Title: What to Preserve and What to Transfer: Faithful, Identity-Preserving Diffusion-based Hairstyle Transfer
• Authors: Chaeyeon Chung, Sunghyun Park, Jeongho Kim, Jaegul Choo,
• Abstract summary: Existing hairstyle transfer approaches rely on StyleGAN, which is pre-trained on cropped and aligned face images. We propose a one-stage hairstyle transfer diffusion model, HairFusion, that applies to real-world scenarios. Our method achieves state-of-the-art performance compared to the existing methods in preserving the integrity of both the transferred hairstyle and the surrounding features.
• Score: 35.80645300182437
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Hairstyle transfer is a challenging task in the image editing field that modifies the hairstyle of a given face image while preserving its other appearance and background features. The existing hairstyle transfer approaches heavily rely on StyleGAN, which is pre-trained on cropped and aligned face images. Hence, they struggle to generalize under challenging conditions such as extreme variations of head poses or focal lengths. To address this issue, we propose a one-stage hairstyle transfer diffusion model, HairFusion, that applies to real-world scenarios. Specifically, we carefully design a hair-agnostic representation as the input of the model, where the original hair information is thoroughly eliminated. Next, we introduce a hair align cross-attention (Align-CA) to accurately align the reference hairstyle with the face image while considering the difference in their face shape. To enhance the preservation of the face image's original features, we leverage adaptive hair blending during the inference, where the output's hair regions are estimated by the cross-attention map in Align-CA and blended with non-hair areas of the face image. Our experimental results show that our method achieves state-of-the-art performance compared to the existing methods in preserving the integrity of both the transferred hairstyle and the surrounding features. The codes are available at https://github.com/cychungg/HairFusion.

Related papers

• HairDiffusion: Vivid Multi-Colored Hair Editing via Latent Diffusion [43.3852998245422]
  We introduce Multi-stage Hairstyle Blend (MHB), effectively separating control of hair color and hairstyle in diffusion latent space. We also train a warping module to align the hair color with the target region. Our method not only tackles the complexity of multi-color hairstyles but also addresses the challenge of preserving original colors.
  arXiv  Detail & Related papers  (2024-10-29T06:51:52Z)
• Realistic and Efficient Face Swapping: A Unified Approach with Diffusion Models [69.50286698375386]
  We propose a novel approach that better harnesses diffusion models for face-swapping. We introduce a mask shuffling technique during inpainting training, which allows us to create a so-called universal model for swapping. Ours is a relatively unified approach and so it is resilient to errors in other off-the-shelf models.
  arXiv  Detail & Related papers  (2024-09-11T13:43:53Z)
• Stable-Hair: Real-World Hair Transfer via Diffusion Model [23.500330976568296]
  Current hair transfer methods struggle to handle diverse and intricate hairstyles, thus limiting their applicability in real-world scenarios. We propose a novel diffusion-based hair transfer framework, named textitStable-Hair, which robustly transfers a wide range of real-world hairstyles onto user-provided faces for virtual hair try-on.
  arXiv  Detail & Related papers  (2024-07-19T07:14:23Z)
• HairFastGAN: Realistic and Robust Hair Transfer with a Fast Encoder-Based Approach [3.737361598712633]
  We present the HairFast model, which achieves high resolution, near real-time performance, and superior reconstruction. Our solution includes a new architecture operating in the FS latent space of StyleGAN. In the most difficult scenario of transferring both shape and color of a hairstyle from different images, our method performs in less than a second on the Nvidia V100.
  arXiv  Detail & Related papers  (2024-04-01T12:59:49Z)
• Portrait Diffusion: Training-free Face Stylization with Chain-of-Painting [64.43760427752532]
  Face stylization refers to the transformation of a face into a specific portrait style. Current methods require the use of example-based adaptation approaches to fine-tune pre-trained generative models. This paper proposes a training-free face stylization framework, named Portrait Diffusion.
  arXiv  Detail & Related papers  (2023-12-03T06:48:35Z)
• Text-Guided Generation and Editing of Compositional 3D Avatars [59.584042376006316]
  Our goal is to create a realistic 3D facial avatar with hair and accessories using only a text description. Existing methods either lack realism, produce unrealistic shapes, or do not support editing.
  arXiv  Detail & Related papers  (2023-09-13T17:59:56Z)
• StyleGAN Salon: Multi-View Latent Optimization for Pose-Invariant Hairstyle Transfer [8.712040236361926]
  The paper seeks to transfer the hairstyle of a reference image to an input photo for virtual hair try-on. We propose a multi-view optimization framework that uses "two different views" of reference composites to semantically guide occluded or ambiguous regions. Our framework produces high-quality results and outperforms prior work in a user study that consists of significantly more challenging hair transfer scenarios.
  arXiv  Detail & Related papers  (2023-04-05T20:49:55Z)
• Style Your Hair: Latent Optimization for Pose-Invariant Hairstyle Transfer via Local-Style-Aware Hair Alignment [29.782276472922398]
  We propose a pose-invariant hairstyle transfer model equipped with latent optimization and a newly presented local-style-matching loss. Our model has strengths in transferring a hairstyle under larger pose differences and preserving local hairstyle textures.
  arXiv  Detail & Related papers  (2022-08-16T14:23:54Z)
• HairFIT: Pose-Invariant Hairstyle Transfer via Flow-based Hair Alignment and Semantic-Region-Aware Inpainting [26.688276902813495]
  We propose a novel framework for pose-invariant hairstyle transfer, HairFIT. Our model consists of two stages: 1) flow-based hair alignment and 2) hair synthesis. Our SIM estimator divides the occluded regions in the source image into different semantic regions to reflect their distinct features during the inpainting.
  arXiv  Detail & Related papers  (2022-06-17T06:55:20Z)
• MichiGAN: Multi-Input-Conditioned Hair Image Generation for Portrait Editing [122.82964863607938]
  MichiGAN is a novel conditional image generation method for interactive portrait hair manipulation. We provide user control over every major hair visual factor, including shape, structure, appearance, and background. We also build an interactive portrait hair editing system that enables straightforward manipulation of hair by projecting intuitive and high-level user inputs.
  arXiv  Detail & Related papers  (2020-10-30T17:59:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.