EmojiHeroVR: A Study on Facial Expression Recognition under Partial Occlusion from Head-Mounted Displays

• URL: http://arxiv.org/abs/2410.03331v1
• Date: Fri, 4 Oct 2024 11:29:04 GMT
• Title: EmojiHeroVR: A Study on Facial Expression Recognition under Partial Occlusion from Head-Mounted Displays
• Authors: Thorben Ortmann, Qi Wang, Larissa Putzar,
• Abstract summary: EmoHeVRDB (EmojiHeroVR Database) includes 3,556 labeled facial images of 1,778 reenacted emotions. EmojiHeVRDB includes data on the activations of 63 facial expressions captured via the Meta Quest Pro VR headset. Best model achieved an accuracy of 69.84% on the test set.
• Score: 4.095418032380801
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Emotion recognition promotes the evaluation and enhancement of Virtual Reality (VR) experiences by providing emotional feedback and enabling advanced personalization. However, facial expressions are rarely used to recognize users' emotions, as Head-Mounted Displays (HMDs) occlude the upper half of the face. To address this issue, we conducted a study with 37 participants who played our novel affective VR game EmojiHeroVR. The collected database, EmoHeVRDB (EmojiHeroVR Database), includes 3,556 labeled facial images of 1,778 reenacted emotions. For each labeled image, we also provide 29 additional frames recorded directly before and after the labeled image to facilitate dynamic Facial Expression Recognition (FER). Additionally, EmoHeVRDB includes data on the activations of 63 facial expressions captured via the Meta Quest Pro VR headset for each frame. Leveraging our database, we conducted a baseline evaluation on the static FER classification task with six basic emotions and neutral using the EfficientNet-B0 architecture. The best model achieved an accuracy of 69.84% on the test set, indicating that FER under HMD occlusion is feasible but significantly more challenging than conventional FER.

Related papers

• ESIQA: Perceptual Quality Assessment of Vision-Pro-based Egocentric Spatial Images [70.68629648595677]
  Egocentric spatial images and videos are emerging as a compelling form of stereoscopic XR content. Egocentric Spatial Images Quality Assessment Database (ESIQAD) is first IQA database dedicated for egocentric spatial images. ESIQAD includes 500 egocentric spatial images, 400 images captured with the Apple Vision Pro and 100 images generated via an iPhone's "Spatial Camera" app.
  arXiv  Detail & Related papers  (2024-07-31T06:20:21Z)
• Facial Emotion Recognition in VR Games [2.5382095320488665]
  We use a Convolutional Neural Network (CNN) to train a model to predict emotions in full-face images where the eyes and eyebrows are covered. The model in these images can accurately recognize seven different emotions which are anger, happiness, disgust, fear, impartiality, sadness and surprise. We analyzed the data collected from our experiment to understand which emotions players experience during the gameplay.
  arXiv  Detail & Related papers  (2023-12-12T01:40:14Z)
• Emotion Separation and Recognition from a Facial Expression by Generating the Poker Face with Vision Transformers [57.1091606948826]
  We propose a novel FER model, named Poker Face Vision Transformer or PF-ViT, to address these challenges. PF-ViT aims to separate and recognize the disturbance-agnostic emotion from a static facial image via generating its corresponding poker face. PF-ViT utilizes vanilla Vision Transformers, and its components are pre-trained as Masked Autoencoders on a large facial expression dataset.
  arXiv  Detail & Related papers  (2022-07-22T13:39:06Z)
• Perceptual Quality Assessment of Omnidirectional Images [81.76416696753947]
  We first establish an omnidirectional IQA (OIQA) database, which includes 16 source images and 320 distorted images degraded by 4 commonly encountered distortion types. Then a subjective quality evaluation study is conducted on the OIQA database in the VR environment. The original and distorted omnidirectional images, subjective quality ratings, and the head and eye movement data together constitute the OIQA database.
  arXiv  Detail & Related papers  (2022-07-06T13:40:38Z)
• EMOCA: Emotion Driven Monocular Face Capture and Animation [59.15004328155593]
  We introduce a novel deep perceptual emotion consistency loss during training, which helps ensure that the reconstructed 3D expression matches the expression depicted in the input image. On the task of in-the-wild emotion recognition, our purely geometric approach is on par with the best image-based methods, highlighting the value of 3D geometry in analyzing human behavior.
  arXiv  Detail & Related papers  (2022-04-24T15:58:35Z)
• Contrastive Context-Aware Learning for 3D High-Fidelity Mask Face Presentation Attack Detection [103.7264459186552]
  Face presentation attack detection (PAD) is essential to secure face recognition systems. Most existing 3D mask PAD benchmarks suffer from several drawbacks. We introduce a largescale High-Fidelity Mask dataset to bridge the gap to real-world applications.
  arXiv  Detail & Related papers  (2021-04-13T12:48:38Z)
• Interpretable Image Emotion Recognition: A Domain Adaptation Approach Using Facial Expressions [11.808447247077902]
  This paper proposes a feature-based domain adaptation technique for identifying emotions in generic images. It addresses the challenge of the limited availability of pre-trained models and well-annotated datasets for Image Emotion Recognition (IER) The proposed IER system demonstrated emotion classification accuracies of 60.98% for the IAPSa dataset, 58.86% for the ArtPhoto dataset, 69.13% for the FI dataset, and 58.06% for the EMOTIC dataset.
  arXiv  Detail & Related papers  (2020-11-17T02:55:16Z)
• Unmasking Communication Partners: A Low-Cost AI Solution for Digitally Removing Head-Mounted Displays in VR-Based Telepresence [62.997667081978825]
  Face-to-face conversation in Virtual Reality (VR) is a challenge when participants wear head-mounted displays (HMD) Past research has shown that high-fidelity face reconstruction with personal avatars in VR is possible under laboratory conditions with high-cost hardware. We propose one of the first low-cost systems for this task which uses only open source, free software and affordable hardware.
  arXiv  Detail & Related papers  (2020-11-06T23:17:12Z)
• Facial Expression Recognition Under Partial Occlusion from Virtual Reality Headsets based on Transfer Learning [0.0]
  convolutional neural network based approaches has become widely adopted due to their proven applicability to Facial Expression Recognition task. However, recognizing facial expression while wearing a head-mounted VR headset is a challenging task due to the upper half of the face being completely occluded. We propose a geometric model to simulate occlusion resulting from a Samsung Gear VR headset that can be applied to existing FER datasets.
  arXiv  Detail & Related papers  (2020-08-12T20:25:07Z)
• Real-time Facial Expression Recognition "In The Wild'' by Disentangling 3D Expression from Identity [6.974241731162878]
  This paper proposes a novel method for human emotion recognition from a single RGB image. We construct a large-scale dataset of facial videos, rich in facial dynamics, identities, expressions, appearance and 3D pose variations. Our proposed framework runs at 50 frames per second and is capable of robustly estimating parameters of 3D expression variation.
  arXiv  Detail & Related papers  (2020-05-12T01:32:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.