Diffusion-Based Makeup Transfer with Facial Region-Aware Makeup Features

• URL: http://arxiv.org/abs/2603.20012v2
• Date: Wed, 25 Mar 2026 11:47:44 GMT
• Title: Diffusion-Based Makeup Transfer with Facial Region-Aware Makeup Features
• Abstract summary: We propose Facial Region-Aware Makeup features (FRAM), which has two stages: (1) makeup CLIP fine-tuning; (2) identity and facial region-aware makeup injection.<n>Specifically, we use learnable tokens to query the makeup CLIP encoder to extract facial region-aware makeup features for makeup injection.<n>The experimental results verify the superiority of our regional controllability and our makeup transfer performance.
• Score: 42.68385892478631
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Current diffusion-based makeup transfer methods commonly use the makeup information encoded by off-the-shelf foundation models (e.g., CLIP) as condition to preserve the makeup style of reference image in the generation. Although effective, these works mainly have two limitations: (1) foundation models pre-trained for generic tasks struggle to capture makeup styles; (2) the makeup features of reference image are injected to the diffusion denoising model as a whole for global makeup transfer, overlooking the facial region-aware makeup features (i.e., eyes, mouth, etc) and limiting the regional controllability for region-specific makeup transfer. To address these, in this work, we propose Facial Region-Aware Makeup features (FRAM), which has two stages: (1) makeup CLIP fine-tuning; (2) identity and facial region-aware makeup injection. For makeup CLIP fine-tuning, unlike prior works using off-the-shelf CLIP, we synthesize annotated makeup style data using GPT-o3 and text-driven image editing model, and then use the data to train a makeup CLIP encoder through self-supervised and image-text contrastive learning. For identity and facial region-aware makeup injection, we construct before-and-after makeup image pairs from the edited images in stage 1 and then use them to learn to inject identity of source image and makeup of reference image to the diffusion denoising model for makeup transfer. Specifically, we use learnable tokens to query the makeup CLIP encoder to extract facial region-aware makeup features for makeup injection, which is learned via an attention loss to enable regional control. As for identity injection, we use a ControlNet Union to encode source image and its 3D mesh simultaneously. The experimental results verify the superiority of our regional controllability and our makeup transfer performance. Code is available at https://github.com/zaczgao/Facial_Region-Aware_Makeup.

Related papers

• Anchoring on Reality: Breaking the Pseudo-Target Ceiling in Makeup Transfer [70.44549081321442]
  Makeup transfer applies a reference cosmetic style to a source face while preserving its identity and geometry.<n>Current methods rely on weak priors or synthetic pseudo-targets from large-scale editing models.<n>We propose Anchoring on Reality Makeup Transfer (ART), a two-stage framework with a reality-anchored refinement cycle.
  arXiv  Detail & Related papers  (2026-06-30T03:28:01Z)
• AvatarMakeup: Realistic Makeup Transfer for 3D Animatable Head Avatars [89.31582684550723]
  AvatarMakeup achieves state-of-the-art makeup transfer quality and consistency throughout animation.<n>Coherent Duplication optimize a global UV map by recoding the averaged facial attributes among the generated makeup images.<n>Experiments demonstrate that AvatarMakeup achieves state-of-the-art makeup transfer quality and consistency throughout animation.
  arXiv  Detail & Related papers  (2025-07-03T08:26:57Z)
• BeautyBank: Encoding Facial Makeup in Latent Space [2.113770213797994]
  We propose BeautyBank, a novel makeup encoder that disentangles pattern features of bare and makeup faces. Our method encodes makeup features into a high-dimensional space, preserving essential details necessary for makeup reconstruction. We also propose a Progressive Makeup Tuning (PMT) strategy, specifically designed to enhance the preservation of detailed makeup features.
  arXiv  Detail & Related papers  (2024-11-18T01:52:31Z)
• Stable-Makeup: When Real-World Makeup Transfer Meets Diffusion Model [15.380297080210559]
  Current makeup transfer methods are limited to simple makeup styles, making them difficult to apply in real-world scenarios.<n>We introduce Stable-Makeup, a novel diffusion-based makeup transfer method capable of robustly transferring a wide range of real-world makeup.
  arXiv  Detail & Related papers  (2024-03-12T15:53:14Z)
• EleGANt: Exquisite and Locally Editable GAN for Makeup Transfer [13.304362849679391]
  We propose Exquisite and locally editable GAN for makeup transfer (EleGANt) It encodes facial attributes into pyramidal feature maps to preserves high-frequency information. EleGANt is the first to achieve customized local editing within arbitrary areas by corresponding editing on the feature maps.
  arXiv  Detail & Related papers  (2022-07-20T11:52:07Z)
• DRAN: Detailed Region-Adaptive Normalization for Conditional Image Synthesis [25.936764522125703]
  We propose a novel normalization module, named Detailed Region-Adaptive Normalization(DRAN) It adaptively learns both fine-grained and coarse-grained style representations. We collect a new makeup dataset (Makeup-Complex dataset) that contains a wide range of complex makeup styles.
  arXiv  Detail & Related papers  (2021-09-29T16:19:37Z)
• PSGAN++: Robust Detail-Preserving Makeup Transfer and Removal [176.47249346856393]
  PSGAN++ is capable of performing both detail-preserving makeup transfer and effective makeup removal. For makeup transfer, PSGAN++ uses a Makeup Distill Network to extract makeup information. For makeup removal, PSGAN++ applies an Identity Distill Network to embed the identity information from with-makeup images into identity matrices.
  arXiv  Detail & Related papers  (2021-05-26T04:37:57Z)
• SOGAN: 3D-Aware Shadow and Occlusion Robust GAN for Makeup Transfer [68.38955698584758]
  We propose a novel makeup transfer method called 3D-Aware Shadow and Occlusion Robust GAN (SOGAN) We first fit a 3D face model and then disentangle the faces into shape and texture. In the texture branch, we map the texture to the UV space and design a UV texture generator to transfer the makeup.
  arXiv  Detail & Related papers  (2021-04-21T14:48:49Z)
• Cosmetic-Aware Makeup Cleanser [109.41917954315784]
  Face verification aims at determining whether a pair of face images belongs to the same identity. Recent studies have revealed the negative impact of facial makeup on the verification performance. This paper proposes a semanticaware makeup cleanser (SAMC) to remove facial makeup under different poses and expressions.
  arXiv  Detail & Related papers  (2020-04-20T09:18:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.