Emotion Separation and Recognition from a Facial Expression by Generating the Poker Face with Vision Transformers

• URL: http://arxiv.org/abs/2207.11081v3
• Date: Fri, 9 Jun 2023 09:12:18 GMT
• Title: Emotion Separation and Recognition from a Facial Expression by Generating the Poker Face with Vision Transformers
• Authors: Jia Li, Jiantao Nie, Dan Guo, Richang Hong, Meng Wang
• Abstract summary: We propose a novel FER model, called Poker Face Vision Transformer or PF-ViT, to separate and recognize the disturbance-agnostic emotion from a static facial image. PF-ViT generates its corresponding poker face without the need for paired images.
• Score: 57.67586172996843
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Representation learning and feature disentanglement have recently attracted much research interests in facial expression recognition. The ubiquitous ambiguity of emotion labels is detrimental to those methods based on conventional supervised representation learning. Meanwhile, directly learning the mapping from a facial expression image to an emotion label lacks explicit supervision signals of facial details. In this paper, we propose a novel FER model, called Poker Face Vision Transformer or PF-ViT, to separate and recognize the disturbance-agnostic emotion from a static facial image via generating its corresponding poker face without the need for paired images. Here, we regard an expressive face as the comprehensive result of a set of facial muscle movements on one's poker face (i.e., emotionless face), inspired by Facial Action Coding System. The proposed PF-ViT leverages vanilla Vision Transformers, and are firstly pre-trained as Masked Autoencoders on a large facial expression dataset without emotion labels, obtaining excellent representations. It mainly consists of five components: 1) an encoder mapping the facial expression to a complete representation, 2) a separator decomposing the representation into an emotional component and an orthogonal residue, 3) a generator that can reconstruct the expressive face and synthesize the poker face, 4) a discriminator distinguishing the fake face produced by the generator, trained adversarially with the encoder and generator, 5) a classification head recognizing the emotion. Quantitative and qualitative results demonstrate the effectiveness of our method, which trumps the state-of-the-art methods on four popular FER testing sets.

Related papers

• Interpretable Explainability in Facial Emotion Recognition and Gamification for Data Collection [0.0]
  Training facial emotion recognition models requires large sets of data and costly annotation processes. We developed a gamified method of acquiring annotated facial emotion data without an explicit labeling effort by humans. We observed significant improvements in the facial emotion perception and expression skills of the players through repeated game play.
  arXiv  Detail & Related papers  (2022-11-09T09:53:48Z)
• PERI: Part Aware Emotion Recognition In The Wild [4.206175795966693]
  This paper focuses on emotion recognition using visual features. We create part aware spatial (PAS) images by extracting key regions from the input image using a mask generated from both body pose and facial landmarks. We provide our results on the publicly available in the wild EMOTIC dataset.
  arXiv  Detail & Related papers  (2022-10-18T20:01:40Z)
• Disentangling Identity and Pose for Facial Expression Recognition [54.50747989860957]
  We propose an identity and pose disentangled facial expression recognition (IPD-FER) model to learn more discriminative feature representation. For identity encoder, a well pre-trained face recognition model is utilized and fixed during training, which alleviates the restriction on specific expression training data. By comparing the difference between synthesized neutral and expressional images of the same individual, the expression component is further disentangled from identity and pose.
  arXiv  Detail & Related papers  (2022-08-17T06:48:13Z)
• Emotion-Controllable Generalized Talking Face Generation [6.22276955954213]
  We propose a one-shot facial geometry-aware emotional talking face generation method. Our method can adapt to arbitrary faces captured in-the-wild by fine-tuning with only a single image of the target identity in neutral emotion.
  arXiv  Detail & Related papers  (2022-05-02T18:41:36Z)
• Learning Fair Face Representation With Progressive Cross Transformer [79.73754444296213]
  We propose a progressive cross transformer (PCT) method for fair face recognition. We show that PCT is capable of mitigating bias in face recognition while achieving state-of-the-art FR performance.
  arXiv  Detail & Related papers  (2021-08-11T01:31:14Z)
• Learning Facial Representations from the Cycle-consistency of Face [23.23272327438177]
  We introduce cycle-consistency in facial characteristics as free supervisory signal to learn facial representations from unlabeled facial images. The learning is realized by superimposing the facial motion cycle-consistency and identity cycle-consistency constraints. Our approach is competitive with those of existing methods, demonstrating the rich and unique information embedded in the disentangled representations.
  arXiv  Detail & Related papers  (2021-08-07T11:30:35Z)
• I Only Have Eyes for You: The Impact of Masks On Convolutional-Based Facial Expression Recognition [78.07239208222599]
  We evaluate how the recently proposed FaceChannel adapts towards recognizing facial expressions from persons with masks. We also perform specific feature-level visualization to demonstrate how the inherent capabilities of the FaceChannel to learn and combine facial features change when in a constrained social interaction scenario.
  arXiv  Detail & Related papers  (2021-04-16T20:03:30Z)
• Human Expression Recognition using Facial Shape Based Fourier Descriptors Fusion [15.063379178217717]
  This paper aims to produce a new facial expression recognition method based on the changes in the facial muscles. The geometric features are used to specify the facial regions i.e., mouth, eyes, and nose. A multi-class support vector machine is applied for classification of seven human expression.
  arXiv  Detail & Related papers  (2020-12-28T05:01:44Z)
• Learning Emotional-Blinded Face Representations [77.7653702071127]
  We propose two face representations that are blind to facial expressions associated to emotional responses. This work is motivated by new international regulations for personal data protection.
  arXiv  Detail & Related papers  (2020-09-18T09:24:10Z)
• Facial Expression Editing with Continuous Emotion Labels [76.36392210528105]
  Deep generative models have achieved impressive results in the field of automated facial expression editing. We propose a model that can be used to manipulate facial expressions in facial images according to continuous two-dimensional emotion labels.
  arXiv  Detail & Related papers  (2020-06-22T13:03:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.