Multimodal Group Emotion Recognition In-the-Wild Towards a Privacy-Safe Non-Individual Approach

• URL: http://arxiv.org/abs/2606.07585v1
• Date: Wed, 27 May 2026 16:36:58 GMT
• Title: Multimodal Group Emotion Recognition In-the-Wild Towards a Privacy-Safe Non-Individual Approach
• Abstract summary: This thesis addresses group emotion recognition (GER) in-the-wild with a focus on privacy preservation.<n>Unlike traditional emotion recognition methods that rely on individual-level cues such as face, gaze, or voice analysis, this work uses collective audio-video signals to infer emotions at the group level.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: This thesis addresses group emotion recognition (GER) in-the-wild with a focus on privacy preservation. Unlike traditional emotion recognition methods that rely on individual-level cues such as face, gaze, or voice analysis, this work uses collective audio-video signals to infer emotions at the group level, reducing risks of individual monitoring and surveillance. Two complementary frameworks are proposed. The first is a cross-attention multimodal architecture for audio-video fusion, combined with Frames Attention Pooling (FAP) for temporal aggregation. It is supported by synthetic data augmentation and validated through ablation studies, demonstrating robustness in real-world GER conditions. The second framework, Variational Encoder Multi-Decoder (VE-MD), learns a shared latent space for emotion classification and structural representation prediction, including body and face cues. Two decoding strategies, DETR-based and heatmap-based, are explored to analyze the role of structural representations in group and individual settings. The thesis makes three main contributions: it clarifies the role of multimodality and structural cues in group-level affective computing; introduces two architectures for privacy-preserving multimodal GER; and shows that competitive performance can be achieved without using individual features as input data.

Related papers

• Enhancing Multimodal Emotion Recognition via Multi-Feature Encoding and Attention-Based Fusion [5.740228163421595]
  This paper proposes a novel multimodal emotion recognition framework.<n>It integrates rich audio and visual feature extraction with an attention-based fusion strategy.<n>Our findings suggest that the proposed framework effectively captures diverse emotional cues from speech and visual expressions.
  arXiv  Detail & Related papers  (2026-09-04T03:40:16Z)
• Bidirectional Learning of Facial Action Units and Expressions via Structured Semantic Mapping across Heterogeneous Datasets [85.74213192818668]
  We propose a Structured Semantic Mapping (SSM) framework for bidirectional AU--FE learning under different data domains and heterogeneous supervision.<n>SSM consists of three key components: (1) a shared visual backbone that learns unified facial representations from dynamic AU and FE videos; (2) semantic mediation via a Textual Semantic Prototype (TSP) module; and (3) a Dynamic Prior Mapping (DPM) module that incorporates prior knowledge derived from the Facial Action Coding System.
  arXiv  Detail & Related papers  (2026-04-12T09:08:32Z)
• Variational Encoder--Multi-Decoder (VE-MD) for Privacy-by-functional-design (Group) Emotion Recognition [0.764671395172401]
  This research proposes VE-MD, a Variational-Multi-Decoder framework for group emotion recognition under a privacy-aware functional design.<n>Rather than providing formal anonymization or cryptographic privacy guarantees, VE-MD is designed to avoid explicit individual monitoring.<n> VE-MD learns a shared latent representation jointly optimized for emotion classification and internal prediction of body and facial structural representations.
  arXiv  Detail & Related papers  (2026-04-02T13:38:29Z)
• Multi-Track Multimodal Learning on iMiGUE: Micro-Gesture and Emotion Recognition [4.909448578374012]
  We present two frameworks designed to tackle both problems on the iMiGUE dataset.<n>For micro-gesture classification, we explore complementary strengths of RGB and 3D pose-based representations.<n>For emotion recognition, our framework extends to behavior-based emotion prediction.
  arXiv  Detail & Related papers  (2025-12-29T08:22:46Z)
• DFR: A Decompose-Fuse-Reconstruct Framework for Multi-Modal Few-Shot Segmentation [21.536784761515968]
  DFR (Decompose, Fuse and Reconstruct) is a novel framework that addresses the challenge of effectively utilizing multi-modal guidance in few-shot segmentation.
  arXiv  Detail & Related papers  (2025-07-22T16:21:32Z)
• CRIA: A Cross-View Interaction and Instance-Adapted Pre-training Framework for Generalizable EEG Representations [52.251569042852815]
  CRIA is an adaptive framework that utilizes variable-length and variable-channel coding to achieve a unified representation of EEG data across different datasets.<n>The model employs a cross-attention mechanism to fuse temporal, spectral, and spatial features effectively.<n> Experimental results on the Temple University EEG corpus and the CHB-MIT dataset show that CRIA outperforms existing methods with the same pre-training conditions.
  arXiv  Detail & Related papers  (2025-06-19T06:31:08Z)
• TAViS: Text-bridged Audio-Visual Segmentation with Foundation Models [123.17643568298116]
  We present TAViS, a novel framework that textbfcouples the knowledge of multimodal foundation models for cross-modal alignment.<n> effectively combining these models poses two key challenges: the difficulty in transferring the knowledge between SAM2 and ImageBind due to their different feature spaces, and the insufficiency of using only segmentation loss for supervision.<n>Our approach achieves superior performance on single-source, multi-source, semantic datasets, and excels in zero-shot settings.
  arXiv  Detail & Related papers  (2025-06-13T03:19:47Z)
• Joint Multimodal Transformer for Emotion Recognition in the Wild [49.735299182004404]
  Multimodal emotion recognition (MMER) systems typically outperform unimodal systems. This paper proposes an MMER method that relies on a joint multimodal transformer (JMT) for fusion with key-based cross-attention.
  arXiv  Detail & Related papers  (2024-03-15T17:23:38Z)
• AMuSE: Adaptive Multimodal Analysis for Speaker Emotion Recognition in Group Conversations [39.79734528362605]
  Multimodal Attention Network captures cross-modal interactions at various levels of spatial abstraction. AMuSE model condenses both spatial and temporal features into two dense descriptors: speaker-level and utterance-level.
  arXiv  Detail & Related papers  (2024-01-26T19:17:05Z)
• Collaboratively Self-supervised Video Representation Learning for Action Recognition [54.92120002380786]
  We design a Collaboratively Self-supervised Video Representation learning framework specific to action recognition.<n>Our method achieves state-of-the-art performance on multiple popular video datasets.
  arXiv  Detail & Related papers  (2024-01-15T10:42:04Z)
• Hierarchical Audio-Visual Information Fusion with Multi-label Joint Decoding for MER 2023 [51.95161901441527]
  In this paper, we propose a novel framework for recognizing both discrete and dimensional emotions. Deep features extracted from foundation models are used as robust acoustic and visual representations of raw video. Our final system achieves state-of-the-art performance and ranks third on the leaderboard on MER-MULTI sub-challenge.
  arXiv  Detail & Related papers  (2023-09-11T03:19:10Z)
• Dual Prototype Attention for Unsupervised Video Object Segmentation [28.725754274542304]
  Unsupervised video object segmentation (VOS) aims to detect and segment the most salient object in videos. This paper proposes two novel prototype-based attention mechanisms, inter-modality attention (IMA) and inter-frame attention (IFA)
  arXiv  Detail & Related papers  (2022-11-22T06:19:17Z)
• Group Gated Fusion on Attention-based Bidirectional Alignment for Multimodal Emotion Recognition [63.07844685982738]
  This paper presents a new model named as Gated Bidirectional Alignment Network (GBAN), which consists of an attention-based bidirectional alignment network over LSTM hidden states. We empirically show that the attention-aligned representations outperform the last-hidden-states of LSTM significantly. The proposed GBAN model outperforms existing state-of-the-art multimodal approaches on the IEMOCAP dataset.
  arXiv  Detail & Related papers  (2022-01-17T09:46:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.