ControlFace: Harnessing Facial Parametric Control for Face Rigging

• URL: http://arxiv.org/abs/2412.01160v4
• Date: Tue, 04 Feb 2025 15:24:15 GMT
• Title: ControlFace: Harnessing Facial Parametric Control for Face Rigging
• Authors: Wooseok Jang, Youngjun Hong, Geonho Cha, Seungryong Kim,
• Abstract summary: We introduce ControlFace, a novel face rigging method conditioned on 3DMM renderings that enables flexible, high-fidelity control. We employ a dual-branch U-Nets: one, referred to as FaceNet, captures identity and fine details, while the other focuses on generation. By training on a facial video dataset, we fully utilize FaceNet's rich representations while ensuring control adherence.
• Score: 31.765503860508378
• License:
• Abstract: Manipulation of facial images to meet specific controls such as pose, expression, and lighting, also known as face rigging, is a complex task in computer vision. Existing methods are limited by their reliance on image datasets, which necessitates individual-specific fine-tuning and limits their ability to retain fine-grained identity and semantic details, reducing practical usability. To overcome these limitations, we introduce ControlFace, a novel face rigging method conditioned on 3DMM renderings that enables flexible, high-fidelity control. We employ a dual-branch U-Nets: one, referred to as FaceNet, captures identity and fine details, while the other focuses on generation. To enhance control precision, the control mixer module encodes the correlated features between the target-aligned control and reference-aligned control, and a novel guidance method, reference control guidance, steers the generation process for better control adherence. By training on a facial video dataset, we fully utilize FaceNet's rich representations while ensuring control adherence. Extensive experiments demonstrate ControlFace's superior performance in identity preservation and control precision, highlighting its practicality. Please see the project website: https://cvlab-kaist.github.io/ControlFace/.

Related papers

• PerLDiff: Controllable Street View Synthesis Using Perspective-Layout Diffusion Models [55.080748327139176]
  We introduce PerLDiff, a method for effective street view image generation that fully leverages perspective 3D geometric information. Our results justify that our PerLDiff markedly enhances the precision of generation on the NuScenes and KITTI datasets.
  arXiv  Detail & Related papers  (2024-07-08T16:46:47Z)
• Face Adapter for Pre-Trained Diffusion Models with Fine-Grained ID and Attribute Control [59.954322727683746]
  Face-Adapter is designed for high-precision and high-fidelity face editing for pre-trained diffusion models. Face-Adapter achieves comparable or even superior performance in terms of motion control precision, ID retention capability, and generation quality.
  arXiv  Detail & Related papers  (2024-05-21T17:50:12Z)
• When ControlNet Meets Inexplicit Masks: A Case Study of ControlNet on its Contour-following Ability [93.15085958220024]
  ControlNet excels at creating content that closely matches precise contours in user-provided masks. When these masks contain noise, as a frequent occurrence with non-expert users, the output would include unwanted artifacts. This paper first highlights the crucial role of controlling the impact of these inexplicit masks with diverse deterioration levels through in-depth analysis. An advanced Shape-aware ControlNet consisting of a deterioration estimator and a shape-prior modulation block is devised.
  arXiv  Detail & Related papers  (2024-03-01T11:45:29Z)
• FineControlNet: Fine-level Text Control for Image Generation with Spatially Aligned Text Control Injection [28.65209293141492]
  FineControlNet provides fine control over each instance's appearance while maintaining the precise pose control capability. FineControlNet achieves superior performance in generating images that follow the user-provided instance-specific text prompts and poses.
  arXiv  Detail & Related papers  (2023-12-14T18:59:43Z)
• DisControlFace: Adding Disentangled Control to Diffusion Autoencoder for One-shot Explicit Facial Image Editing [14.537856326925178]
  We focus on exploring explicit fine-grained control of generative facial image editing. We propose a novel diffusion-based editing framework, named DisControlFace. Our model can be trained using 2D in-the-wild portrait images without requiring 3D or video data.
  arXiv  Detail & Related papers  (2023-12-11T08:16:55Z)
• Fine-grained Controllable Video Generation via Object Appearance and Context [74.23066823064575]
  We propose fine-grained controllable video generation (FACTOR) to achieve detailed control. FACTOR aims to control objects' appearances and context, including their location and category. Our method achieves controllability of object appearances without finetuning, which reduces the per-subject optimization efforts for the users.
  arXiv  Detail & Related papers  (2023-12-05T17:47:33Z)
• Reference-Guided Large-Scale Face Inpainting with Identity and Texture Control [4.866431869728018]
  Face inpainting aims at plausibly predicting missing pixels of face images within a corrupted region. Most existing methods rely on generative models learning a face image distribution from a big dataset. We propose a novel reference-guided face inpainting method that fills the large-scale missing region with identity and texture control guided by a reference face image.
  arXiv  Detail & Related papers  (2023-03-13T11:22:37Z)
• Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation [96.66010515343106]
  We propose a clean yet effective framework to generate pose-controllable talking faces. We operate on raw face images, using only a single photo as an identity reference. Our model has multiple advanced capabilities including extreme view robustness and talking face frontalization.
  arXiv  Detail & Related papers  (2021-04-22T15:10:26Z)
• FaceController: Controllable Attribute Editing for Face in the Wild [74.56117807309576]
  We propose a simple feed-forward network to generate high-fidelity manipulated faces. By simply employing some existing and easy-obtainable prior information, our method can control, transfer, and edit diverse attributes of faces in the wild. In our method, we decouple identity, expression, pose, and illumination using 3D priors; separate texture and colors by using region-wise style codes.
  arXiv  Detail & Related papers  (2021-02-23T02:47:28Z)
• GAN-Control: Explicitly Controllable GANs [4.014524824655106]
  We present a framework for training GANs with explicit control over generated images. We are able to control the generated image by settings exact attributes such as age, pose, expression, etc.
  arXiv  Detail & Related papers  (2021-01-07T10:54:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.