Emo3D: Metric and Benchmarking Dataset for 3D Facial Expression Generation from Emotion Description

• URL: http://arxiv.org/abs/2410.02049v1
• Date: Wed, 2 Oct 2024 21:31:24 GMT
• Title: Emo3D: Metric and Benchmarking Dataset for 3D Facial Expression Generation from Emotion Description
• Authors: Mahshid Dehghani, Amirahmad Shafiee, Ali Shafiei, Neda Fallah, Farahmand Alizadeh, Mohammad Mehdi Gholinejad, Hamid Behroozi, Jafar Habibi, Ehsaneddin Asgari,
• Abstract summary: "Emo3D" is an extensive "Text-Image-Expression dataset" spanning a wide spectrum of human emotions. We generate a diverse array of textual descriptions, facilitating the capture of a broad spectrum of emotional expressions. "Emo3D" has great applications in animation design, virtual reality, and emotional human-computer interaction.
• Score: 3.52270271101496
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Existing 3D facial emotion modeling have been constrained by limited emotion classes and insufficient datasets. This paper introduces "Emo3D", an extensive "Text-Image-Expression dataset" spanning a wide spectrum of human emotions, each paired with images and 3D blendshapes. Leveraging Large Language Models (LLMs), we generate a diverse array of textual descriptions, facilitating the capture of a broad spectrum of emotional expressions. Using this unique dataset, we conduct a comprehensive evaluation of language-based models' fine-tuning and vision-language models like Contranstive Language Image Pretraining (CLIP) for 3D facial expression synthesis. We also introduce a new evaluation metric for this task to more directly measure the conveyed emotion. Our new evaluation metric, Emo3D, demonstrates its superiority over Mean Squared Error (MSE) metrics in assessing visual-text alignment and semantic richness in 3D facial expressions associated with human emotions. "Emo3D" has great applications in animation design, virtual reality, and emotional human-computer interaction.

Related papers

• MMHead: Towards Fine-grained Multi-modal 3D Facial Animation [68.04052669266174]
  We construct a large-scale multi-modal 3D facial animation dataset, MMHead. MMHead consists of 49 hours of 3D facial motion sequences, speech audios, and rich hierarchical text annotations. Based on the MMHead dataset, we establish benchmarks for two new tasks: text-induced 3D talking head animation and text-to-3D facial motion generation.
  arXiv  Detail & Related papers  (2024-10-10T09:37:01Z)
• EmoTalk3D: High-Fidelity Free-View Synthesis of Emotional 3D Talking Head [30.138347111341748]
  We present a novel approach for synthesizing 3D talking heads with controllable emotion. Our model enables controllable emotion in the generated talking heads and can be rendered in wide-range views. Experiments demonstrate the effectiveness of our approach in generating high-fidelity and emotion-controllable 3D talking heads.
  arXiv  Detail & Related papers  (2024-08-01T05:46:57Z)
• EmoVOCA: Speech-Driven Emotional 3D Talking Heads [12.161006152509653]
  We propose an innovative data-driven technique for creating a synthetic dataset, called EmoVOCA. We then designed and trained an emotional 3D talking head generator that accepts a 3D face, an audio file, an emotion label, and an intensity value as inputs, and learns to animate the audio-synchronized lip movements with expressive traits of the face.
  arXiv  Detail & Related papers  (2024-03-19T16:33:26Z)
• Emotion Rendering for Conversational Speech Synthesis with Heterogeneous Graph-Based Context Modeling [50.99252242917458]
  Conversational Speech Synthesis (CSS) aims to accurately express an utterance with the appropriate prosody and emotional inflection within a conversational setting. To address the issue of data scarcity, we meticulously create emotional labels in terms of category and intensity. Our model outperforms the baseline models in understanding and rendering emotions.
  arXiv  Detail & Related papers  (2023-12-19T08:47:50Z)
• EmoSet: A Large-scale Visual Emotion Dataset with Rich Attributes [53.95428298229396]
  We introduce EmoSet, the first large-scale visual emotion dataset annotated with rich attributes. EmoSet comprises 3.3 million images in total, with 118,102 of these images carefully labeled by human annotators. Motivated by psychological studies, in addition to emotion category, each image is also annotated with a set of describable emotion attributes.
  arXiv  Detail & Related papers  (2023-07-16T06:42:46Z)
• EmoTalk: Speech-Driven Emotional Disentanglement for 3D Face Animation [28.964917860664492]
  Speech-driven 3D face animation aims to generate realistic facial expressions that match the speech content and emotion. This paper proposes an end-to-end neural network to disentangle different emotions in speech so as to generate rich 3D facial expressions. Our approach outperforms state-of-the-art methods and exhibits more diverse facial movements.
  arXiv  Detail & Related papers  (2023-03-20T13:22:04Z)
• HUMANISE: Language-conditioned Human Motion Generation in 3D Scenes [54.61610144668777]
  We present a novel scene-and-language conditioned generative model that can produce 3D human motions in 3D scenes. Our experiments demonstrate that our model generates diverse and semantically consistent human motions in 3D scenes.
  arXiv  Detail & Related papers  (2022-10-18T10:14:11Z)
• EMOCA: Emotion Driven Monocular Face Capture and Animation [59.15004328155593]
  We introduce a novel deep perceptual emotion consistency loss during training, which helps ensure that the reconstructed 3D expression matches the expression depicted in the input image. On the task of in-the-wild emotion recognition, our purely geometric approach is on par with the best image-based methods, highlighting the value of 3D geometry in analyzing human behavior.
  arXiv  Detail & Related papers  (2022-04-24T15:58:35Z)
• Enhancing Cognitive Models of Emotions with Representation Learning [58.2386408470585]
  We present a novel deep learning-based framework to generate embedding representations of fine-grained emotions. Our framework integrates a contextualized embedding encoder with a multi-head probing model. Our model is evaluated on the Empathetic Dialogue dataset and shows the state-of-the-art result for classifying 32 emotions.
  arXiv  Detail & Related papers  (2021-04-20T16:55:15Z)
• Real-time Facial Expression Recognition "In The Wild'' by Disentangling 3D Expression from Identity [6.974241731162878]
  This paper proposes a novel method for human emotion recognition from a single RGB image. We construct a large-scale dataset of facial videos, rich in facial dynamics, identities, expressions, appearance and 3D pose variations. Our proposed framework runs at 50 frames per second and is capable of robustly estimating parameters of 3D expression variation.
  arXiv  Detail & Related papers  (2020-05-12T01:32:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.