Variational Encoder--Multi-Decoder (VE-MD) for Privacy-by-functional-design (Group) Emotion Recognition

• URL: http://arxiv.org/abs/2604.02397v1
• Date: Thu, 02 Apr 2026 13:38:29 GMT
• Title: Variational Encoder--Multi-Decoder (VE-MD) for Privacy-by-functional-design (Group) Emotion Recognition
• Abstract summary: This research proposes VE-MD, a Variational-Multi-Decoder framework for group emotion recognition under a privacy-aware functional design.<n>Rather than providing formal anonymization or cryptographic privacy guarantees, VE-MD is designed to avoid explicit individual monitoring.<n> VE-MD learns a shared latent representation jointly optimized for emotion classification and internal prediction of body and facial structural representations.
• Score: 0.764671395172401
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Group Emotion Recognition (GER) aims to infer collective affect in social environments such as classrooms, crowds, and public events. Many existing approaches rely on explicit individual-level processing, including cropped faces, person tracking, or per-person feature extraction, which makes the analysis pipeline person-centric and raises privacy concerns in deployment scenarios where only group-level understanding is needed. This research proposes VE-MD, a Variational Encoder-Multi-Decoder framework for group emotion recognition under a privacy-aware functional design. Rather than providing formal anonymization or cryptographic privacy guarantees, VE-MD is designed to avoid explicit individual monitoring by constraining the model to predict only aggregate group-level affect, without identity recognition or per-person emotion outputs. VE-MD learns a shared latent representation jointly optimized for emotion classification and internal prediction of body and facial structural representations. Two structural decoding strategies are investigated: a transformer-based PersonQuery decoder and a dense Heatmap decoder that naturally accommodates variable group sizes. Experiments on six in-the-wild datasets, including two GER and four Individual Emotion Recognition (IER) benchmarks, show that structural supervision consistently improves representation learning. More importantly, the results reveal a clear distinction between GER and IER: optimizing the latent space alone is often insufficient for GER because it tends to attenuate interaction-related cues, whereas preserving explicit structural outputs improves collective affect inference. In contrast, projected structural representations seem to act as an effective denoising bottleneck for IER. VE-MD achieves state-of-the-art performance on GAF-3.0 (up to 90.06%) and VGAF (82.25% with multimodal fusion with audio). These results show that preserving interaction-related structural information is particularly beneficial for group-level affect modeling without relying on prior individual feature extraction. On IER datasets using multimodal fusion with audio modality, VE-MD outperforms SOTA on SamSemo (77.9%, adding text modality) while achieving competitive performances on MER-MULTI (63.8%), DFEW (70.7%) and EngageNet (69.0).

Related papers

• Multimodal Group Emotion Recognition In-the-Wild Towards a Privacy-Safe Non-Individual Approach [0.0]
  This thesis addresses group emotion recognition (GER) in-the-wild with a focus on privacy preservation.<n>Unlike traditional emotion recognition methods that rely on individual-level cues such as face, gaze, or voice analysis, this work uses collective audio-video signals to infer emotions at the group level.
  arXiv  Detail & Related papers  (2026-05-27T16:36:58Z)
• Personalized Cross-Modal Emotional Correlation Learning for Speech-Preserving Facial Expression Manipulation [63.90351637818903]
  Speech-preserving facial expression manipulation aims to enhance human expressiveness without altering mouth movements tied to the original speech.<n>Current Visual-Language Models (VLMs) can extract aligned visual and semantic features, making them a promising source of supervision.<n>We propose a Personalized Cross-Modal Emotional Correlation Learning algorithm that refines VLM-based supervision.
  arXiv  Detail & Related papers  (2026-04-28T06:02:23Z)
• AffectAgent: Collaborative Multi-Agent Reasoning for Retrieval-Augmented Multimodal Emotion Recognition [62.16431420189863]
  LLM-based multimodal emotion recognition relies on static parametric memory and often hallucinates when interpreting nuanced affective states.<n>We introduce AffectAgent, an affect-oriented multi-agent retrieval-augmented generation framework.<n>AffectAgent comprises three jointly optimized agents, namely a query planner, an evidence filter, and an emotion generator.
  arXiv  Detail & Related papers  (2026-04-14T13:49:19Z)
• Brainprint-Modulated Target Speaker Extraction [1.6974371408199849]
  We propose Brainprint-ulated Speaker Extraction (BM-TSE), a novel framework for personalized and high-fidelity extraction.<n>The core of our framework is a personalized modulation mechanism, where a unified brainmap embedding is learned.<n>BM-TSE achieves state-of-the-art performance, significantly outperforming existing methods.
  arXiv  Detail & Related papers  (2025-09-22T15:17:35Z)
• CoBEVMoE: Heterogeneity-aware Feature Fusion with Dynamic Mixture-of-Experts for Collaborative Perception [21.27356211403264]
  We propose a novel collaborative perception framework that operates in the Bird's Eye View (BEV) space.<n>We introduce a Dynamic Expert Metric Loss (DEML) to enhance inter-expert diversity and improve the discriminability of the fused representation.
  arXiv  Detail & Related papers  (2025-09-21T14:56:05Z)
• Joint Learning using Mixture-of-Expert-Based Representation for Enhanced Speech Generation and Robust Emotion Recognition [54.44798086835314]
  Speech emotion recognition (SER) plays a critical role in building emotion-aware speech systems, but its performance degrades significantly under noisy conditions.<n>We propose the Sparse Mixture-of-Experts Representation Integration Technique (Sparse MERIT), a flexible MTL framework that applies frame-wise expert routing over self-supervised speech representations.<n> Experiments on the MSP-Podcast corpus show that Sparse MERIT consistently outperforms baseline models on both SER and SE tasks.
  arXiv  Detail & Related papers  (2025-09-10T10:18:56Z)
• CRIA: A Cross-View Interaction and Instance-Adapted Pre-training Framework for Generalizable EEG Representations [52.251569042852815]
  CRIA is an adaptive framework that utilizes variable-length and variable-channel coding to achieve a unified representation of EEG data across different datasets.<n>The model employs a cross-attention mechanism to fuse temporal, spectral, and spatial features effectively.<n> Experimental results on the Temple University EEG corpus and the CHB-MIT dataset show that CRIA outperforms existing methods with the same pre-training conditions.
  arXiv  Detail & Related papers  (2025-06-19T06:31:08Z)
• AMuSE: Adaptive Multimodal Analysis for Speaker Emotion Recognition in Group Conversations [39.79734528362605]
  Multimodal Attention Network captures cross-modal interactions at various levels of spatial abstraction. AMuSE model condenses both spatial and temporal features into two dense descriptors: speaker-level and utterance-level.
  arXiv  Detail & Related papers  (2024-01-26T19:17:05Z)
• Self-Supervised Neuron Segmentation with Multi-Agent Reinforcement Learning [53.00683059396803]
  Mask image model (MIM) has been widely used due to its simplicity and effectiveness in recovering original information from masked images. We propose a decision-based MIM that utilizes reinforcement learning (RL) to automatically search for optimal image masking ratio and masking strategy. Our approach has a significant advantage over alternative self-supervised methods on the task of neuron segmentation.
  arXiv  Detail & Related papers  (2023-10-06T10:40:46Z)
• Exploiting Modality-Specific Features For Multi-Modal Manipulation Detection And Grounding [54.49214267905562]
  We construct a transformer-based framework for multi-modal manipulation detection and grounding tasks. Our framework simultaneously explores modality-specific features while preserving the capability for multi-modal alignment. We propose an implicit manipulation query (IMQ) that adaptively aggregates global contextual cues within each modality.
  arXiv  Detail & Related papers  (2023-09-22T06:55:41Z)
• Object Segmentation by Mining Cross-Modal Semantics [68.88086621181628]
  We propose a novel approach by mining the Cross-Modal Semantics to guide the fusion and decoding of multimodal features. Specifically, we propose a novel network, termed XMSNet, consisting of (1) all-round attentive fusion (AF), (2) coarse-to-fine decoder (CFD), and (3) cross-layer self-supervision.
  arXiv  Detail & Related papers  (2023-05-17T14:30:11Z)
• Adversarial Deep Feature Extraction Network for User Independent Human Activity Recognition [4.988898367111902]
  We present an adversarial subject-independent feature extraction method with the maximum mean discrepancy (MMD) regularization for human activity recognition. We evaluate the method on well-known public data sets showing that it significantly improves user-independent performance and reduces variance in results.
  arXiv  Detail & Related papers  (2021-10-23T07:50:32Z)
• DexDeepFM: Ensemble Diversity Enhanced Extreme Deep Factorization Machine Model [8.73107818888638]
  An ensemble diversity enhanced extreme deep factorization machine model (DexDeepFM) is proposed. Experiments on two public real-world datasets show the superiority of the proposed model.
  arXiv  Detail & Related papers  (2021-04-05T14:06:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.