How Do You Perceive My Face? Recognizing Facial Expressions in Multi-Modal Context by Modeling Mental Representations

• URL: http://arxiv.org/abs/2409.02566v1
• Date: Wed, 4 Sep 2024 09:32:40 GMT
• Title: How Do You Perceive My Face? Recognizing Facial Expressions in Multi-Modal Context by Modeling Mental Representations
• Authors: Florian Blume, Runfeng Qu, Pia Bideau, Martin Maier, Rasha Abdel Rahman, Olaf Hellwich,
• Abstract summary: We introduce a novel approach for facial expression classification that goes beyond simple classification tasks. Our model accurately classifies a perceived face and synthesizes the corresponding mental representation perceived by a human when observing a face in context. We evaluate synthesized expressions in a human study, showing that our model effectively produces approximations of human mental representations.
• Score: 5.895694050664867
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Facial expression perception in humans inherently relies on prior knowledge and contextual cues, contributing to efficient and flexible processing. For instance, multi-modal emotional context (such as voice color, affective text, body pose, etc.) can prompt people to perceive emotional expressions in objectively neutral faces. Drawing inspiration from this, we introduce a novel approach for facial expression classification that goes beyond simple classification tasks. Our model accurately classifies a perceived face and synthesizes the corresponding mental representation perceived by a human when observing a face in context. With this, our model offers visual insights into its internal decision-making process. We achieve this by learning two independent representations of content and context using a VAE-GAN architecture. Subsequently, we propose a novel attention mechanism for context-dependent feature adaptation. The adapted representation is used for classification and to generate a context-augmented expression. We evaluate synthesized expressions in a human study, showing that our model effectively produces approximations of human mental representations. We achieve State-of-the-Art classification accuracies of 81.01% on the RAVDESS dataset and 79.34% on the MEAD dataset. We make our code publicly available.

Related papers

• Beyond Vision: How Large Language Models Interpret Facial Expressions from Valence-Arousal Values [6.987852837732702]
  Large Language Models primarily operate through text-based inputs and outputs, yet human emotion is communicated through both verbal and non-verbal cues, including facial expressions. This study investigates whether LLMs can infer affective meaning from dimensions of facial expressions-Valence and Arousal values, structured numerical representations, rather than using raw visual input.
  arXiv  Detail & Related papers  (2025-02-08T09:54:03Z)
• Probing the contents of semantic representations from text, behavior, and brain data using the psychNorms metabase [0.0]
  We evaluate the similarities and differences between semantic representations derived from text, behavior, and brain data. We establish behavior as an important complement to text for capturing human representations and behavior.
  arXiv  Detail & Related papers  (2024-12-06T10:44:20Z)
• Knowledge-Enhanced Facial Expression Recognition with Emotional-to-Neutral Transformation [66.53435569574135]
  Existing facial expression recognition methods typically fine-tune a pre-trained visual encoder using discrete labels. We observe that the rich knowledge in text embeddings, generated by vision-language models, is a promising alternative for learning discriminative facial expression representations. We propose a novel knowledge-enhanced FER method with an emotional-to-neutral transformation.
  arXiv  Detail & Related papers  (2024-09-13T07:28:57Z)
• ST-Gait++: Leveraging spatio-temporal convolutions for gait-based emotion recognition on videos [3.1489012476109854]
  We propose a framework for emotion recognition through the analysis of gait. Our model is composed of a sequence of spatial-temporal Graph Convolutional Networks. We evaluate our proposed framework on the E-Gait dataset, composed of a total of 2177 samples.
  arXiv  Detail & Related papers  (2024-05-22T18:24:21Z)
• Emotion Rendering for Conversational Speech Synthesis with Heterogeneous Graph-Based Context Modeling [50.99252242917458]
  Conversational Speech Synthesis (CSS) aims to accurately express an utterance with the appropriate prosody and emotional inflection within a conversational setting. To address the issue of data scarcity, we meticulously create emotional labels in terms of category and intensity. Our model outperforms the baseline models in understanding and rendering emotions.
  arXiv  Detail & Related papers  (2023-12-19T08:47:50Z)
• CIAO! A Contrastive Adaptation Mechanism for Non-Universal Facial Expression Recognition [80.07590100872548]
  We propose Contrastive Inhibitory Adaptati On (CIAO), a mechanism that adapts the last layer of facial encoders to depict specific affective characteristics on different datasets. CIAO presents an improvement in facial expression recognition performance over six different datasets with very unique affective representations.
  arXiv  Detail & Related papers  (2022-08-10T15:46:05Z)
• Seeking Subjectivity in Visual Emotion Distribution Learning [93.96205258496697]
  Visual Emotion Analysis (VEA) aims to predict people's emotions towards different visual stimuli. Existing methods often predict visual emotion distribution in a unified network, neglecting the inherent subjectivity in its crowd voting process. We propose a novel textitSubjectivity Appraise-and-Match Network (SAMNet) to investigate the subjectivity in visual emotion distribution.
  arXiv  Detail & Related papers  (2022-07-25T02:20:03Z)
• Enhancing Cognitive Models of Emotions with Representation Learning [58.2386408470585]
  We present a novel deep learning-based framework to generate embedding representations of fine-grained emotions. Our framework integrates a contextualized embedding encoder with a multi-head probing model. Our model is evaluated on the Empathetic Dialogue dataset and shows the state-of-the-art result for classifying 32 emotions.
  arXiv  Detail & Related papers  (2021-04-20T16:55:15Z)
• A Multi-resolution Approach to Expression Recognition in the Wild [9.118706387430883]
  We propose a multi-resolution approach to solve the Facial Expression Recognition task. We ground our intuition on the observation that often faces images are acquired at different resolutions. To our aim, we use a ResNet-like architecture, equipped with Squeeze-and-Excitation blocks, trained on the Affect-in-the-Wild 2 dataset.
  arXiv  Detail & Related papers  (2021-03-09T21:21:02Z)
• Facial Expression Editing with Continuous Emotion Labels [76.36392210528105]
  Deep generative models have achieved impressive results in the field of automated facial expression editing. We propose a model that can be used to manipulate facial expressions in facial images according to continuous two-dimensional emotion labels.
  arXiv  Detail & Related papers  (2020-06-22T13:03:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.