SEREP: Semantic Facial Expression Representation for Robust In-the-Wild Capture and Retargeting

• URL: http://arxiv.org/abs/2412.14371v2
• Date: Fri, 20 Dec 2024 21:57:01 GMT
• Title: SEREP: Semantic Facial Expression Representation for Robust In-the-Wild Capture and Retargeting
• Authors: Arthur Josi, Luiz Gustavo Hafemann, Abdallah Dib, Emeline Got, Rafael M. O. Cruz, Marc-Andre Carbonneau,
• Abstract summary: We propose SEREP (Semantic Expression Representation), a model that disentangles expression from identity at the semantic level. We train a model to predict expression from monocular images using a novel semi-supervised scheme that relies on domain adaptation. Our experiments show that SEREP outperforms state-of-the-art methods, capturing challenging expressions and transferring them to novel identities.
• Score: 4.083283519300837
• License:
• Abstract: Monocular facial performance capture in-the-wild is challenging due to varied capture conditions, face shapes, and expressions. Most current methods rely on linear 3D Morphable Models, which represent facial expressions independently of identity at the vertex displacement level. We propose SEREP (Semantic Expression Representation), a model that disentangles expression from identity at the semantic level. It first learns an expression representation from unpaired 3D facial expressions using a cycle consistency loss. Then we train a model to predict expression from monocular images using a novel semi-supervised scheme that relies on domain adaptation. In addition, we introduce MultiREX, a benchmark addressing the lack of evaluation resources for the expression capture task. Our experiments show that SEREP outperforms state-of-the-art methods, capturing challenging expressions and transferring them to novel identities.

Related papers

• OSDFace: One-Step Diffusion Model for Face Restoration [72.5045389847792]
  Diffusion models have demonstrated impressive performance in face restoration. We propose OSDFace, a novel one-step diffusion model for face restoration. Results demonstrate that OSDFace surpasses current state-of-the-art (SOTA) methods in both visual quality and quantitative metrics.
  arXiv  Detail & Related papers  (2024-11-26T07:07:48Z)
• A Generative Framework for Self-Supervised Facial Representation Learning [18.094262972295702]
  Self-supervised representation learning has gained increasing attention for strong generalization ability without relying on paired datasets. Self-supervised facial representation learning remains unsolved due to the coupling of facial identities, expressions, and external factors like pose and light. We propose LatentFace, a novel generative framework for self-supervised facial representations.
  arXiv  Detail & Related papers  (2023-09-15T09:34:05Z)
• GaFET: Learning Geometry-aware Facial Expression Translation from In-The-Wild Images [55.431697263581626]
  We introduce a novel Geometry-aware Facial Expression Translation framework, which is based on parametric 3D facial representations and can stably decoupled expression. We achieve higher-quality and more accurate facial expression transfer results compared to state-of-the-art methods, and demonstrate applicability of various poses and complex textures.
  arXiv  Detail & Related papers  (2023-08-07T09:03:35Z)
• Controllable 3D Generative Adversarial Face Model via Disentangling Shape and Appearance [63.13801759915835]
  3D face modeling has been an active area of research in computer vision and computer graphics. This paper proposes a new 3D face generative model that can decouple identity and expression.
  arXiv  Detail & Related papers  (2022-08-30T13:40:48Z)
• ImFace: A Nonlinear 3D Morphable Face Model with Implicit Neural Representations [21.389170615787368]
  This paper presents a novel 3D morphable face model, namely ImFace, to learn a nonlinear and continuous space with implicit neural representations. It builds two explicitly disentangled deformation fields to model complex shapes associated with identities and expressions, respectively, and designs an improved learning strategy to extend embeddings of expressions. In addition to ImFace, an effective preprocessing pipeline is proposed to address the issue of watertight input requirement in implicit representations.
  arXiv  Detail & Related papers  (2022-03-28T05:37:59Z)
• Learning to Aggregate and Personalize 3D Face from In-the-Wild Photo Collection [65.92058628082322]
  Non-parametric face modeling aims to reconstruct 3D face only from images without shape assumptions. This paper presents a novel Learning to Aggregate and Personalize framework for unsupervised robust 3D face modeling.
  arXiv  Detail & Related papers  (2021-06-15T03:10:17Z)
• MorphGAN: One-Shot Face Synthesis GAN for Detecting Recognition Bias [13.162012586770576]
  We describe a simulator that applies specific head pose and facial expression adjustments to images of previously unseen people. We show that by augmenting small datasets of faces with new poses and expressions improves the recognition performance by up to 9% depending on the augmentation and data scarcity.
  arXiv  Detail & Related papers  (2020-12-09T18:43:03Z)
• Learning an Animatable Detailed 3D Face Model from In-The-Wild Images [50.09971525995828]
  We present the first approach to jointly learn a model with animatable detail and a detailed 3D face regressor from in-the-wild images. Our DECA model is trained to robustly produce a UV displacement map from a low-dimensional latent representation. We introduce a novel detail-consistency loss to disentangle person-specific details and expression-dependent wrinkles.
  arXiv  Detail & Related papers  (2020-12-07T19:30:45Z)
• LEED: Label-Free Expression Editing via Disentanglement [57.09545215087179]
  LEED framework is capable of editing the expression of both frontal and profile facial images without requiring any expression label. Two novel losses are designed for optimal expression disentanglement and consistent synthesis.
  arXiv  Detail & Related papers  (2020-07-17T13:36:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.