Occlusion-Robust Multimodal Emotion Recognition in VR via Fusion of Facial Images and EMG

• URL: http://arxiv.org/abs/2609.03569v1
• Date: Thu, 03 Sep 2026 09:17:32 GMT
• Title: Occlusion-Robust Multimodal Emotion Recognition in VR via Fusion of Facial Images and EMG
• Abstract summary: Head-mounted displays (HMDs) fundamentally limit emotion recognition in virtual reality (VR)<n>We address this challenge by fusing lower-face video with facial electromyography (EMG) from the occluded upper face to classify seven emotional categories (six basic emotions plus neutral)<n>We introduce a synchronized multimodal dataset from 20 participants, pairing lower-face video with seven-channel upper-face EMG elicited by validated emotion stimuli.
• Score: 8.520211823360826
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Head-mounted displays (HMDs) fundamentally limit emotion recognition in virtual reality (VR): by occluding the upper face, they render conventional image-based facial expression analysis incomplete, particularly for applications requiring real-time affective assessment. We address this challenge by fusing lower-face video with facial electromyography (EMG) from the occluded upper face to classify seven emotional categories (six basic emotions plus neutral). We introduce a synchronized multimodal dataset from 20 participants, pairing lower-face video with seven-channel upper-face EMG elicited by validated emotion stimuli. Under subject-independent test, our proposed late-fusion architecture merging convolutional visual embeddings with RBF-kernel EMG representations achieves 51% macro-F1, outperforming both image-only (41%) and EMG-only (43%) baselines. These results demonstrate that upper-face EMG provides robust complementary information under HMD-induced visual occlusion and establish a foundation for multimodal emotion recognition in naturalistic VR environments. This approach facilitates affect-adaptive applications, including communication training and therapeutic interventions. The dataset will be shared upon request under an ethical-use agreement.

Related papers

• What Does the Brain See? Multiview Neural Representations to Demystify the Brain-Visual Alignment [16.54359569394021]
  We introduce a unified multiview EEG representation learning framework for aligning brain responses with visual semantic embeddings.<n>We show that our method achieves state-of-the-art performance, with 54.8% Top-1 and 85.6% Top-5 accuracy in the within-subject setting and 15.3% Top-1 and 45.4% Top-5 accuracy in the cross-subject setting.
  arXiv  Detail & Related papers  (2026-06-24T11:33:21Z)
• Adaptive Physical-Facial Representation Fusion via Subject-Invariant Cross-Modal Prompt Tuning for Video-Based Emotion Recognition [14.850489141041017]
  We propose a cross-modal cross-tuning prompt-tuning method for video-based emotion recognition.<n>The proposed method consistently outperforms strong baselines in both recognition accuracy and generalization ability.
  arXiv  Detail & Related papers  (2026-05-07T05:33:31Z)
• Smile on the Face, Sadness in the Eyes: Bridging the Emotion Gap with a Multimodal Dataset of Eye and Facial Behaviors [49.833812625518554]
  We introduce eye behaviors as an important emotional cue and construct an Eye-behavior-aided Multimodal Emotion Recognition dataset.<n>In the experiment, we introduce seven multimodal benchmark protocols for a variety of comprehensive evaluations of the EMER dataset.<n>The results show that the EMERT outperforms other state-of-the-art multimodal methods by a great margin, revealing the importance of modeling eye behaviors for robust ER.
  arXiv  Detail & Related papers  (2025-12-18T12:52:55Z)
• FACE: Few-shot Adapter with Cross-view Fusion for Cross-subject EEG Emotion Recognition [57.08108545219043]
  Cross-subject EEG emotion recognition is challenged by significant inter-subject variability and intricately entangled intra-subject variability.<n>Recent few-shot learning paradigms attempt to address these limitations but often encounter catastrophic overfitting during subject-specific adaptation with limited samples.<n>This article introduces the few-shot adapter with a cross-view fusion method called FACE for cross-subject EEG emotion recognition.
  arXiv  Detail & Related papers  (2025-03-24T03:16:52Z)
• Unimodal and Multimodal Static Facial Expression Recognition for Virtual Reality Users with EmoHeVRDB [4.095418032380801]
  We explored the potential of utilizing Facial Expression Activations (FEAs) captured via the Meta Quest Pro Virtual Reality (VR) headset for Facial Expression Recognition (FER) in VR settings.<n>We compared several unimodal approaches and achieved up to 73.02% accuracy for the static FER task with seven emotion categories.<n>We integrated FEA and image data in multimodal approaches, observing significant improvements in recognition accuracy.
  arXiv  Detail & Related papers  (2024-12-15T20:49:46Z)
• Smile upon the Face but Sadness in the Eyes: Emotion Recognition based on Facial Expressions and Eye Behaviors [63.194053817609024]
  We introduce eye behaviors as an important emotional cues for the creation of a new Eye-behavior-aided Multimodal Emotion Recognition dataset. For the first time, we provide annotations for both Emotion Recognition (ER) and Facial Expression Recognition (FER) in the EMER dataset. We specifically design a new EMERT architecture to concurrently enhance performance in both ER and FER.
  arXiv  Detail & Related papers  (2024-11-08T04:53:55Z)
• EmojiHeroVR: A Study on Facial Expression Recognition under Partial Occlusion from Head-Mounted Displays [4.095418032380801]
  EmoHeVRDB (EmojiHeroVR Database) includes 3,556 labeled facial images of 1,778 reenacted emotions. EmojiHeVRDB includes data on the activations of 63 facial expressions captured via the Meta Quest Pro VR headset. Best model achieved an accuracy of 69.84% on the test set.
  arXiv  Detail & Related papers  (2024-10-04T11:29:04Z)
• Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning [55.127202990679976]
  We introduce the MERR dataset, containing 28,618 coarse-grained and 4,487 fine-grained annotated samples across diverse emotional categories. This dataset enables models to learn from varied scenarios and generalize to real-world applications. We propose Emotion-LLaMA, a model that seamlessly integrates audio, visual, and textual inputs through emotion-specific encoders.
  arXiv  Detail & Related papers  (2024-06-17T03:01:22Z)
• Emotion Separation and Recognition from a Facial Expression by Generating the Poker Face with Vision Transformers [57.1091606948826]
  We propose a novel FER model, named Poker Face Vision Transformer or PF-ViT, to address these challenges. PF-ViT aims to separate and recognize the disturbance-agnostic emotion from a static facial image via generating its corresponding poker face. PF-ViT utilizes vanilla Vision Transformers, and its components are pre-trained as Masked Autoencoders on a large facial expression dataset.
  arXiv  Detail & Related papers  (2022-07-22T13:39:06Z)
• Interpretable Image Emotion Recognition: A Domain Adaptation Approach Using Facial Expressions [11.808447247077902]
  This paper proposes a feature-based domain adaptation technique for identifying emotions in generic images.<n>It addresses the challenge of the limited availability of pre-trained models and well-annotated datasets for Image Emotion Recognition (IER)<n>The proposed IER system demonstrated emotion classification accuracies of 61.86% for the IAPSa dataset, 62.47 for the ArtPhoto dataset, 70.78% for the FI dataset, and 59.72% for the EMOTIC dataset.
  arXiv  Detail & Related papers  (2020-11-17T02:55:16Z)
• Multi-Scale Thermal to Visible Face Verification via Attribute Guided Synthesis [55.29770222566124]
  We use attributes extracted from visible images to synthesize attribute-preserved visible images from thermal imagery for cross-modal matching. A novel multi-scale generator is proposed to synthesize the visible image from the thermal image guided by the extracted attributes. A pre-trained VGG-Face network is leveraged to extract features from the synthesized image and the input visible image for verification.
  arXiv  Detail & Related papers  (2020-04-20T01:45:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.