MUPA$^{2}$E: Multimodal Unified Perception with Asymmetric Attention for Emotion Assessment

• URL: http://arxiv.org/abs/2608.15999v1
• Date: Mon, 17 Aug 2026 01:26:47 GMT
• Title: MUPA$^{2}$E: Multimodal Unified Perception with Asymmetric Attention for Emotion Assessment
• Abstract summary: This paper presents MUPAtextsuperscript2E, a unified perception framework that processes facial video and electroencephalography (EEG) through a single shared asymmetric-attention backbone.<n> Facial video is represented through axis-folded frame tokens, while EEG is processed either as a raw multichannel waveform or projected into the spatial domain for multimodal fusion.<n>Using the original recordings, with shorter trials zero-padded to match the longest duration, merged fusion at stride$30$ achieves the highest validation performance and a test accuracy of $70.07%$.
• Score: 4.682152272555128
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Automatic emotion assessment can benefit from combining neural and behavioral signals, but many multimodal approaches rely on separate, modality-specific feature-extraction pipelines before fusion. This paper presents MUPA\textsuperscript{2}E, a unified perception framework that processes facial video and electroencephalography (EEG) through a single shared asymmetric-attention backbone. Facial video is represented through axis-folded frame tokens, while EEG is processed either as a raw multichannel waveform or projected into the spatial domain for multimodal fusion. The framework is evaluated on the DMER dataset under a stratified subject-independent protocol, comparing unimodal video, unimodal EEG, and fused video--EEG configurations with per-channel and merged EEG projections. Using the original recordings, with shorter trials zero-padded to match the longest duration, merged fusion at stride~$30$ achieves the highest validation performance and a test accuracy of $70.07\%$. Further analysis revealed that recording duration is unevenly distributed across the affective classes, making the padding pattern a potential classification cue. Controlling for this factor by cropping all recordings to a common duration of $20$ seconds yielded a test accuracy of $62.71\%$, providing a stricter duration-controlled assessment of the framework in which differences in recording length are removed as a potential classification cue. These findings demonstrate the feasibility of processing structurally different neural and visual signals within a compact unified architecture while highlighting the importance of controlling duration-related cues in affective datasets.

Related papers

• Knowledge-Guided Multimodal Reasoning over Interacting Streams for Video-Level Ambivalence and Hesitancy Recognition [0.9297355862757838]
  PRISM-AH is a framework that treats A/H as a multimodal conflict that unfolds over time.<n>A knowledge-guided large language model then reasons over structured evidence using the expert cue taxonomy of the dataset.<n>On the labelled public test partition of 525 videos, PRISM-AH attains a macro F1 of 0.6133, compared to the reported zero-shot baseline of 0.2827.
  arXiv  Detail & Related papers  (2026-07-28T16:44:38Z)
• Nuanced Emotion Recognition Based on a Segment-based MLLM Framework Leveraging Qwen3-Omni for AH Detection [15.9620440408998]
  This paper proposes a recognition framework that integrates temporal segment modeling with Multimodal Large Language Models.<n>We employ a segment-based strategy, partitioning videos into short clips with a maximum duration of 5 seconds.<n> Experimental results demonstrate that the proposed method achieves an accuracy of 85.1% on the test set.
  arXiv  Detail & Related papers  (2026-03-12T08:31:42Z)
• Complementarity-Supervised Spectral-Band Routing for Multimodal Emotion Recognition [60.20529806857076]
  Multimodal emotion recognition fuses cues such as text, video, and audio to understand individual emotional states.<n>Prior methods face two main limitations: mechanically relying on independent unimodal performance, and coarse-grained fusion conflicting with the fine-grained representations required by emotion tasks.<n>We propose the Complementarity-Supervised Multi-Band Expert Network, named Atsuko, to model fine-grained complementary features via multi-scale band decomposition and expert collaboration.
  arXiv  Detail & Related papers  (2026-03-07T03:58:48Z)
• Cross-Temporal Attention Fusion (CTAF) for Multimodal Physiological Signals in Self-Supervised Learning [0.0]
  Cross-Temporal Attention Fusion (CTAF) is a self-supervised module that learns soft bidirectional alignments between modalities.<n>CTAF yields higher cosine margins for matched pairs and better cross-modal token retrieval within one second.<n>Our contributions are a time-aware fusion mechanism that directly models correspondence, an alignment-driven self-supervised objective tailored to EEG and physiology, and an evaluation protocol that measures alignment quality itself.
  arXiv  Detail & Related papers  (2026-02-02T20:40:04Z)
• Digital FAST: An AI-Driven Multimodal Framework for Rapid and Early Stroke Screening [0.7136933021609076]
  This study presents a fast, non-invasive multimodal deep learning framework for automatic binary stroke screening based on data collected during the F.A.S.T. assessment.<n>The proposed approach integrates complementary information from facial expressions, speech signals, and upper-body movements to enhance diagnostic robustness.
  arXiv  Detail & Related papers  (2026-01-17T03:35:39Z)
• AHDMIL: Asymmetric Hierarchical Distillation Multi-Instance Learning for Fast and Accurate Whole-Slide Image Classification [51.525891360380285]
  AHDMIL is an Asymmetric Hierarchical Distillation Multi-Instance Learning framework.<n>It eliminates irrelevant patches through a two-step training process.<n>It consistently outperforms previous state-of-the-art methods in both classification performance and inference speed.
  arXiv  Detail & Related papers  (2025-08-07T07:47:16Z)
• EEG-SCMM: Soft Contrastive Masked Modeling for Cross-Corpus EEG-Based Emotion Recognition [0.862468061241377]
  We propose a novel framework termed Soft Contrastive Masked Modeling (SCMM) to tackle the challenge of cross-corpus EEG-based emotion recognition.<n>SCMM integrates soft contrastive learning with a hybrid masking strategy to effectively capture emotion dynamics.<n>In experiments, SCMM achieves an average accuracy of 4.26% under both same-class and different-class cross-corpus settings.
  arXiv  Detail & Related papers  (2024-08-17T12:35:13Z)
• Joint Multimodal Transformer for Emotion Recognition in the Wild [49.735299182004404]
  Multimodal emotion recognition (MMER) systems typically outperform unimodal systems. This paper proposes an MMER method that relies on a joint multimodal transformer (JMT) for fusion with key-based cross-attention.
  arXiv  Detail & Related papers  (2024-03-15T17:23:38Z)
• Revisiting the Evaluation of Image Synthesis with GANs [55.72247435112475]
  This study presents an empirical investigation into the evaluation of synthesis performance, with generative adversarial networks (GANs) as a representative of generative models. In particular, we make in-depth analyses of various factors, including how to represent a data point in the representation space, how to calculate a fair distance using selected samples, and how many instances to use from each set.
  arXiv  Detail & Related papers  (2023-04-04T17:54:32Z)
• Multiple Time Series Fusion Based on LSTM An Application to CAP A Phase Classification Using EEG [56.155331323304]
  Deep learning based electroencephalogram channels' feature level fusion is carried out in this work. Channel selection, fusion, and classification procedures were optimized by two optimization algorithms.
  arXiv  Detail & Related papers  (2021-12-18T14:17:49Z)
• Generalizing electrocardiogram delineation: training convolutional neural networks with synthetic data augmentation [63.51064808536065]
  Existing databases for ECG delineation are small, being insufficient in size and in the array of pathological conditions they represent. This article delves has two main contributions. First, a pseudo-synthetic data generation algorithm was developed, based in probabilistically composing ECG traces given "pools" of fundamental segments, as cropped from the original databases, and a set of rules for their arrangement into coherent synthetic traces. Second, two novel segmentation-based loss functions have been developed, which attempt at enforcing the prediction of an exact number of independent structures and at producing closer segmentation boundaries by focusing on a reduced number of samples.
  arXiv  Detail & Related papers  (2021-11-25T10:11:41Z)
• $M^3$T: Multi-Modal Continuous Valence-Arousal Estimation in the Wild [86.40973759048957]
  This report describes a multi-modal multi-task ($M3$T) approach underlying our submission to the valence-arousal estimation track of the Affective Behavior Analysis in-the-wild (ABAW) Challenge. In the proposed $M3$T framework, we fuse both visual features from videos and acoustic features from the audio tracks to estimate the valence and arousal. We evaluated the $M3$T framework on the validation set provided by ABAW and it significantly outperforms the baseline method.
  arXiv  Detail & Related papers  (2020-02-07T18:53:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.