UF-AMA: A unified framework for cross-domain emotion recognition via adaptive multimodal alignment

• URL: http://arxiv.org/abs/2606.00170v1
• Date: Fri, 29 May 2026 12:56:42 GMT
• Title: UF-AMA: A unified framework for cross-domain emotion recognition via adaptive multimodal alignment
• Abstract summary: We propose a Unified Framework with Adaptive Multimodal Alignment (UF-AMA) to address cross-subject and cross-session emotion recognition.<n>UF-AMA achieves state-of-the-art (SOTA) performance in both cross-subject and cross-session tasks.
• Score: 9.928876781046894
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In recent years, emotion recognition based on physiological signals such as electroencephalogram (EEG) has gained considerable attention, as internal physiological data offer greater objectivity and reliability compared to external behavioral data like facial expressions. However, due to distribution shifts caused by individual and contextual differences, along with variations in sample quality across modalities, constructing a cross-domain multimodal emotion recognition model with high generalization and robustness remains a key challenge. In this study, we propose a Unified Framework with Adaptive Multimodal Alignment (UF-AMA) to address cross-subject and cross-session emotion recognition using multimodal physiological signals. First, we construct a cross-modal feature fusion network comprising Transformer encoders and multi-head cross-attention modules, enabling the deep integration of EEG signals and eye-tracking data. Subsequently, we introduce a confidence-aware screening mechanism that dynamically assesses the predictive reliability of each modality branch on target domain samples, partitions samples into different quality subsets, and accordingly applies global consistency alignment and cross-modal distillation. Finally, we propose a multi-level domain adaptation framework that jointly optimizes the marginal and conditional distributions of both local modality-specific and global fusion features, thereby reducing cross-domain distribution shifts at multiple granularities. Extensive experiments on the SEED and SEED-IV datasets demonstrate that UF-AMA achieves state-of-the-art (SOTA) performance in both cross-subject and cross-session tasks. The source code is available at: https://github.com/BetterCoderLab/UF-AMA.

Related papers

• SuppreSensing: Expert-Guided Feature Recalibration and Discrepancy Augmentation for Multimodal Object Detection [19.446402308279676]
  We propose SuppreSensing, which reformulates multimodal fusion as a selective collaboration process.<n>SupreSensing first designs an Expert-driven Multimodal Feature Recalibration (EMFR) module.<n>A modality-specific augmentation strategy is employed to attribute specific modality features.<n>Experiments on the DroneVehicle and VEDAI datasets demonstrate that SuppreSensing achieves state-of-the-art detection performance.
  arXiv  Detail & Related papers  (2026-08-21T10:06:00Z)
• AffectAgent: Collaborative Multi-Agent Reasoning for Retrieval-Augmented Multimodal Emotion Recognition [62.16431420189863]
  LLM-based multimodal emotion recognition relies on static parametric memory and often hallucinates when interpreting nuanced affective states.<n>We introduce AffectAgent, an affect-oriented multi-agent retrieval-augmented generation framework.<n>AffectAgent comprises three jointly optimized agents, namely a query planner, an evidence filter, and an emotion generator.
  arXiv  Detail & Related papers  (2026-04-14T13:49:19Z)
• MOSAIC: Multi-Domain Orthogonal Session Adaptive Intent Capture for Prescient Recommendations [0.0]
  Capturing user intent across heterogeneous behavioral domains is a challenge in session-based recommender systems.<n>We propose MOSAIC, a Multi-Domain Orthogonal Session Adaptive Intent Capture framework.<n>We conduct experiments on two large-scale real-world benchmarks spanning multiple domains and interaction types.
  arXiv  Detail & Related papers  (2026-04-11T10:33:12Z)
• Local-Global Feature Fusion for Subject-Independent EEG Emotion Recognition [5.248014945077116]
  We propose a fusion framework that integrates (i) local, channel-wise descriptors and (ii) global, trial-level descriptors.<n>Experiments under a leave-one-subject-out protocol demonstrate that the proposed method consistently outperforms single-view and classical baselines.
  arXiv  Detail & Related papers  (2026-01-13T00:27:01Z)
• DAMSDAN: Distribution-Aware Multi-Source Domain Adaptation Network for Cross-Domain EEG-based Emotion Recognition [19.010493629153288]
  We propose a distribution-aware multi-source domain adaptation network (DAMSDAN) for emotion recognition.<n>DAMSDAN integrates prototype-based constraints with adversarial learning to drive the encoder toward discriminative, domain-invariant emotion representations.<n>Experiments on SEED and SEED-IV show average accuracies of 94.86% and 79.78% for cross-subject, and 95.12% and 83.15% for cross-session protocols.
  arXiv  Detail & Related papers  (2025-10-20T12:18:46Z)
• Multisource Collaborative Domain Generalization for Cross-Scene Remote Sensing Image Classification [57.945437355714155]
  Cross-scene image classification aims to transfer prior knowledge of ground materials to annotate regions with different distributions.<n>Existing approaches focus on single-source domain generalization to unseen target domains.<n>We propose a novel multi-source collaborative domain generalization framework (MS-CDG) based on homogeneity and heterogeneity characteristics of multi-source remote sensing data.
  arXiv  Detail & Related papers  (2024-12-05T06:15:08Z)
• MMA-DFER: MultiModal Adaptation of unimodal models for Dynamic Facial Expression Recognition in-the-wild [81.32127423981426]
  Multimodal emotion recognition based on audio and video data is important for real-world applications. Recent methods have focused on exploiting advances of self-supervised learning (SSL) for pre-training of strong multimodal encoders. We propose a different perspective on the problem and investigate the advancement of multimodal DFER performance by adapting SSL-pre-trained disjoint unimodal encoders.
  arXiv  Detail & Related papers  (2024-04-13T13:39:26Z)
• Joint Multimodal Transformer for Emotion Recognition in the Wild [49.735299182004404]
  Multimodal emotion recognition (MMER) systems typically outperform unimodal systems. This paper proposes an MMER method that relies on a joint multimodal transformer (JMT) for fusion with key-based cross-attention.
  arXiv  Detail & Related papers  (2024-03-15T17:23:38Z)
• Unified Frequency-Assisted Transformer Framework for Detecting and Grounding Multi-Modal Manipulation [109.1912721224697]
  We present the Unified Frequency-Assisted transFormer framework, named UFAFormer, to address the DGM4 problem. By leveraging the discrete wavelet transform, we decompose images into several frequency sub-bands, capturing rich face forgery artifacts. Our proposed frequency encoder, incorporating intra-band and inter-band self-attentions, explicitly aggregates forgery features within and across diverse sub-bands.
  arXiv  Detail & Related papers  (2023-09-18T11:06:42Z)
• TACOformer:Token-channel compounded Cross Attention for Multimodal Emotion Recognition [0.951828574518325]
  We propose a comprehensive perspective of multimodal fusion that integrates channel-level and token-level cross-modal interactions. Specifically, we introduce a unified cross attention module called Token-chAnnel COmpound (TACO) Cross Attention. We also propose a 2D position encoding method to preserve information about the spatial distribution of EEG signal channels.
  arXiv  Detail & Related papers  (2023-06-23T16:28:12Z)
• Object Segmentation by Mining Cross-Modal Semantics [68.88086621181628]
  We propose a novel approach by mining the Cross-Modal Semantics to guide the fusion and decoding of multimodal features. Specifically, we propose a novel network, termed XMSNet, consisting of (1) all-round attentive fusion (AF), (2) coarse-to-fine decoder (CFD), and (3) cross-layer self-supervision.
  arXiv  Detail & Related papers  (2023-05-17T14:30:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.