Are 3D Face Shapes Expressive Enough for Recognising Continuous Emotions and Action Unit Intensities?

• URL: http://arxiv.org/abs/2207.01113v2
• Date: Sat, 27 May 2023 05:45:16 GMT
• Title: Are 3D Face Shapes Expressive Enough for Recognising Continuous Emotions and Action Unit Intensities?
• Authors: Mani Kumar Tellamekala, \"Omer S\"umer, Bj\"orn W. Schuller, Elisabeth Andr\'e, Timo Giesbrecht, Michel Valstar
• Abstract summary: This work focuses on a promising alternative based on parametric 3D face shape alignment models. We benchmark four recent 3D face alignment models: ExpNet, 3DDFA-V2, DECA, and EMOCA. We report that 3D face features were on par with 2D appearance features in AUs 4, 6, 10, 12, and 25, but not the entire set of AUs.
• Score: 1.2233362977312945
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recognising continuous emotions and action unit (AU) intensities from face videos requires a spatial and temporal understanding of expression dynamics. Existing works primarily rely on 2D face appearances to extract such dynamics. This work focuses on a promising alternative based on parametric 3D face shape alignment models, which disentangle different factors of variation, including expression-induced shape variations. We aim to understand how expressive 3D face shapes are in estimating valence-arousal and AU intensities compared to the state-of-the-art 2D appearance-based models. We benchmark four recent 3D face alignment models: ExpNet, 3DDFA-V2, DECA, and EMOCA. In valence-arousal estimation, expression features of 3D face models consistently surpassed previous works and yielded an average concordance correlation of .739 and .574 on SEWA and AVEC 2019 CES corpora, respectively. We also study how 3D face shapes performed on AU intensity estimation on BP4D and DISFA datasets, and report that 3D face features were on par with 2D appearance features in AUs 4, 6, 10, 12, and 25, but not the entire set of AUs. To understand this discrepancy, we conduct a correspondence analysis between valence-arousal and AUs, which points out that accurate prediction of valence-arousal may require the knowledge of only a few AUs.

Related papers

• Ig3D: Integrating 3D Face Representations in Facial Expression Inference [12.975434103690812]
  This study aims to investigate the impacts of integrating 3D representations into the facial expression inference task. We first assess the performance of two 3D face representations (both based on the 3D morphable model, FLAME) for the FEI tasks. We then explore two fusion architectures, intermediate fusion and late fusion, for integrating the 3D face representations with existing 2D inference frameworks. Our proposed method outperforms the state-of-the-art AffectNet VA estimation and RAF-DB classification tasks.
  arXiv  Detail & Related papers  (2024-08-29T21:08:07Z)
• AnimateMe: 4D Facial Expressions via Diffusion Models [72.63383191654357]
  Recent advances in diffusion models have enhanced the capabilities of generative models in 2D animation. We employ Graph Neural Networks (GNNs) as denoising diffusion models in a novel approach, formulating the diffusion process directly on the mesh space. This facilitates the generation of facial deformations through a mesh-diffusion-based model.
  arXiv  Detail & Related papers  (2024-03-25T21:40:44Z)
• From Static to Dynamic: Adapting Landmark-Aware Image Models for Facial Expression Recognition in Videos [88.08209394979178]
  Dynamic facial expression recognition (DFER) in the wild is still hindered by data limitations. We introduce a novel Static-to-Dynamic model (S2D) that leverages existing SFER knowledge and dynamic information implicitly encoded in extracted facial landmark-aware features.
  arXiv  Detail & Related papers  (2023-12-09T03:16:09Z)
• ImFace++: A Sophisticated Nonlinear 3D Morphable Face Model with Implicit Neural Representations [25.016000421755162]
  This paper presents a novel 3D morphable face model, named ImFace++, to learn a sophisticated and continuous space with implicit neural representations. ImFace++ first constructs two explicitly disentangled deformation fields to model complex shapes associated with identities and expressions. A refinement displacement field within the template space is further incorporated, enabling fine-grained learning of individual-specific facial details.
  arXiv  Detail & Related papers  (2023-12-07T03:53:53Z)
• Decaf: Monocular Deformation Capture for Face and Hand Interactions [77.75726740605748]
  This paper introduces the first method that allows tracking human hands interacting with human faces in 3D from single monocular RGB videos. We model hands as articulated objects inducing non-rigid face deformations during an active interaction. Our method relies on a new hand-face motion and interaction capture dataset with realistic face deformations acquired with a markerless multi-view camera system.
  arXiv  Detail & Related papers  (2023-09-28T17:59:51Z)
• Controllable 3D Generative Adversarial Face Model via Disentangling Shape and Appearance [63.13801759915835]
  3D face modeling has been an active area of research in computer vision and computer graphics. This paper proposes a new 3D face generative model that can decouple identity and expression.
  arXiv  Detail & Related papers  (2022-08-30T13:40:48Z)
• RiCS: A 2D Self-Occlusion Map for Harmonizing Volumetric Objects [68.85305626324694]
  Ray-marching in Camera Space (RiCS) is a new method to represent the self-occlusions of foreground objects in 3D into a 2D self-occlusion map. We show that our representation map not only allows us to enhance the image quality but also to model temporally coherent complex shadow effects.
  arXiv  Detail & Related papers  (2022-05-14T05:35:35Z)
• ImFace: A Nonlinear 3D Morphable Face Model with Implicit Neural Representations [21.389170615787368]
  This paper presents a novel 3D morphable face model, namely ImFace, to learn a nonlinear and continuous space with implicit neural representations. It builds two explicitly disentangled deformation fields to model complex shapes associated with identities and expressions, respectively, and designs an improved learning strategy to extend embeddings of expressions. In addition to ImFace, an effective preprocessing pipeline is proposed to address the issue of watertight input requirement in implicit representations.
  arXiv  Detail & Related papers  (2022-03-28T05:37:59Z)
• Identity-Expression Ambiguity in 3D Morphable Face Models [5.38250259923059]
  We show that non-orthogonality of the variation in identity and expression can cause identity-expression ambiguity in 3D Morphable Models. We demonstrate this effect with 3D shapes directly as well as through an inverse rendering task.
  arXiv  Detail & Related papers  (2021-09-29T06:11:43Z)
• Magnifying Subtle Facial Motions for Effective 4D Expression Recognition [56.806738404887824]
  The flow of 3D faces is first analyzed to capture the spatial deformations. The obtained temporal evolution of these deformations are fed into a magnification method. The latter, main contribution of this paper, allows revealing subtle (hidden) deformations which enhance the emotion classification performance.
  arXiv  Detail & Related papers  (2021-05-05T20:47:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.