A Generalist FaceX via Learning Unified Facial Representation

• URL: http://arxiv.org/abs/2401.00551v1
• Date: Sun, 31 Dec 2023 17:41:48 GMT
• Title: A Generalist FaceX via Learning Unified Facial Representation
• Authors: Yue Han, Jiangning Zhang, Junwei Zhu, Xiangtai Li, Yanhao Ge, Wei Li, Chengjie Wang, Yong Liu, Xiaoming Liu, Ying Tai
• Abstract summary: FaceX is a novel facial generalist model capable of handling diverse facial tasks simultaneously. Our versatile FaceX achieves competitive performance compared to elaborate task-specific models on popular facial editing tasks.
• Score: 77.74407008931486
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This work presents FaceX framework, a novel facial generalist model capable of handling diverse facial tasks simultaneously. To achieve this goal, we initially formulate a unified facial representation for a broad spectrum of facial editing tasks, which macroscopically decomposes a face into fundamental identity, intra-personal variation, and environmental factors. Based on this, we introduce Facial Omni-Representation Decomposing (FORD) for seamless manipulation of various facial components, microscopically decomposing the core aspects of most facial editing tasks. Furthermore, by leveraging the prior of a pretrained StableDiffusion (SD) to enhance generation quality and accelerate training, we design Facial Omni-Representation Steering (FORS) to first assemble unified facial representations and then effectively steer the SD-aware generation process by the efficient Facial Representation Controller (FRC). %Without any additional features, Our versatile FaceX achieves competitive performance compared to elaborate task-specific models on popular facial editing tasks. Full codes and models will be available at https://github.com/diffusion-facex/FaceX.

Related papers

• FaceXFormer: A Unified Transformer for Facial Analysis [59.94066615853198]
  FaceXformer is an end-to-end unified transformer model for a range of facial analysis tasks. Our model effectively handles images "in-the-wild," demonstrating its robustness and generalizability across eight different tasks.
  arXiv  Detail & Related papers  (2024-03-19T17:58:04Z)
• Self-Supervised Facial Representation Learning with Facial Region Awareness [13.06996608324306]
  Self-supervised pre-training has been proven to be effective in learning transferable representations that benefit various visual tasks. Recent efforts toward this goal are limited to treating each face image as a whole. We propose a novel self-supervised facial representation learning framework to learn consistent global and local facial representations.
  arXiv  Detail & Related papers  (2024-03-04T15:48:56Z)
• End-to-end Face-swapping via Adaptive Latent Representation Learning [12.364688530047786]
  This paper proposes a novel and end-to-end integrated framework for high resolution and attribute preservation face swapping. Our framework integrating facial perceiving and blending into the end-to-end training and testing process can achieve high realistic face-swapping on wild faces.
  arXiv  Detail & Related papers  (2023-03-07T19:16:20Z)
• DiffFace: Diffusion-based Face Swapping with Facial Guidance [24.50570533781642]
  We propose a diffusion-based face swapping framework for the first time, called DiffFace. It is composed of training ID conditional DDPM, sampling with facial guidance, and a target-preserving blending. DiffFace achieves better benefits such as training stability, high fidelity, diversity of the samples, and controllability.
  arXiv  Detail & Related papers  (2022-12-27T02:51:46Z)
• Emotion Separation and Recognition from a Facial Expression by Generating the Poker Face with Vision Transformers [57.1091606948826]
  We propose a novel FER model, named Poker Face Vision Transformer or PF-ViT, to address these challenges. PF-ViT aims to separate and recognize the disturbance-agnostic emotion from a static facial image via generating its corresponding poker face. PF-ViT utilizes vanilla Vision Transformers, and its components are pre-trained as Masked Autoencoders on a large facial expression dataset.
  arXiv  Detail & Related papers  (2022-07-22T13:39:06Z)
• FaceFormer: Scale-aware Blind Face Restoration with Transformers [18.514630131883536]
  We propose a novel scale-aware blind face restoration framework, named FaceFormer, which formulates facial feature restoration as scale-aware transformation. Our proposed method trained with synthetic dataset generalizes better to a natural low quality images than current state-of-the-arts.
  arXiv  Detail & Related papers  (2022-07-20T10:08:34Z)
• Pro-UIGAN: Progressive Face Hallucination from Occluded Thumbnails [53.080403912727604]
  We propose a multi-stage Progressive Upsampling and Inpainting Generative Adversarial Network, dubbed Pro-UIGAN. It exploits facial geometry priors to replenish and upsample (8*) the occluded and tiny faces. Pro-UIGAN achieves visually pleasing HR faces, reaching superior performance in downstream tasks.
  arXiv  Detail & Related papers  (2021-08-02T02:29:24Z)
• DotFAN: A Domain-transferred Face Augmentation Network for Pose and Illumination Invariant Face Recognition [94.96686189033869]
  We propose a 3D model-assisted domain-transferred face augmentation network (DotFAN) DotFAN can generate a series of variants of an input face based on the knowledge distilled from existing rich face datasets collected from other domains. Experiments show that DotFAN is beneficial for augmenting small face datasets to improve their within-class diversity.
  arXiv  Detail & Related papers  (2020-02-23T08:16:34Z)
• Face Hallucination with Finishing Touches [65.14864257585835]
  We present a novel Vivid Face Hallucination Generative Adversarial Network (VividGAN) for simultaneously super-resolving and frontalizing tiny non-frontal face images. VividGAN consists of coarse-level and fine-level Face Hallucination Networks (FHnet) and two discriminators, i.e., Coarse-D and Fine-D. Experiments demonstrate that our VividGAN achieves photo-realistic frontal HR faces, reaching superior performance in downstream tasks.
  arXiv  Detail & Related papers  (2020-02-09T07:33:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.