Adaptive Physical-Facial Representation Fusion via Subject-Invariant Cross-Modal Prompt Tuning for Video-Based Emotion Recognition

• URL: http://arxiv.org/abs/2605.05694v1
• Date: Thu, 07 May 2026 05:33:31 GMT
• Title: Adaptive Physical-Facial Representation Fusion via Subject-Invariant Cross-Modal Prompt Tuning for Video-Based Emotion Recognition
• Abstract summary: We propose a cross-modal cross-tuning prompt-tuning method for video-based emotion recognition.<n>The proposed method consistently outperforms strong baselines in both recognition accuracy and generalization ability.
• Score: 14.850489141041017
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Emotion recognition from facial videos enables non-contact inference of human emotional states. Although facial expressions are widely used cues, they cannot fully reflect intrinsic affective states. Remote photoplethysmography (rPPG) provides complementary physiological information, but it is highly susceptible to noise and inter-subject variability, limiting generalization to unseen individuals. Existing multimodal methods combine facial and rPPG features, yet their fusion strategies often disrupt pretrained facial representations and lack explicit mechanisms to suppress subject-specific variations. To address these issues, we propose a subject-invariant cross-modal prompt-tuning framework for video-based emotion recognition. Specifically, rPPG waveforms are transformed into noise-robust time-frequency representations (TFRs), from which modality-complementary prompts are generated to modulate facial tokens within a frozen Vision Transformer (ViT). This design enables effective cross-modal interaction while preserving the generalizable facial representations learned by the pretrained backbone. In addition, we introduce a decoupled shared-specific adapter (DSSA) into each ViT layer to explicitly separate subject-shared and subject-specific components, thereby improving cross-subject generalization. Experiments on the MAHNOB-HCI and DEAP benchmarks demonstrate that the proposed method consistently outperforms strong baselines in both recognition accuracy and generalization ability, highlighting its effectiveness for video-based emotion recognition.

Related papers

• SVF-CR: Synchronized Visual-Facial Cross-Refinement for Multimodal Ambivalence and Hesitancy Recognition [4.562539324528659]
  Ambivalence and hesitancy are subtle behavioral states that are expressed through a combination of verbal content, facial behavior, visual context, and acoustic cues.<n>We propose a synchronized visual-facial cross-refinement framework (SVF-CR) with pairwise multimodal evidence fusion for ambivalence and hesitancy recognition.
  arXiv  Detail & Related papers  (2026-07-10T13:48:35Z)
• Personalized Cross-Modal Emotional Correlation Learning for Speech-Preserving Facial Expression Manipulation [63.90351637818903]
  Speech-preserving facial expression manipulation aims to enhance human expressiveness without altering mouth movements tied to the original speech.<n>Current Visual-Language Models (VLMs) can extract aligned visual and semantic features, making them a promising source of supervision.<n>We propose a Personalized Cross-Modal Emotional Correlation Learning algorithm that refines VLM-based supervision.
  arXiv  Detail & Related papers  (2026-04-28T06:02:23Z)
• Multimodal Emotion Recognition via Bi-directional Cross-Attention and Temporal Modeling [2.8037951156321377]
  We propose a multimodal emotion recognition framework for the Expression task in the 10th Affective Behavior Analysis in-the-wild (ABAW) Challenge.<n>Our framework builds on large-scale pre-trained models for visual and audio representation learning and integrates them in a unified multimodal architecture.<n> Experimental results on the ABAW 10th EXPR benchmark demonstrate the effectiveness of the proposed framework.
  arXiv  Detail & Related papers  (2026-03-12T14:20:29Z)
• Text-guided Weakly Supervised Framework for Dynamic Facial Expression Recognition [49.41688891301643]
  Dynamic facial expression recognition aims to identify emotional states by modeling the temporal changes in facial movements across video sequences.<n>A key challenge in DFER is the many-to-one labeling problem, where a video composed of numerous frames is assigned a single emotion label.<n>We propose TG-DFER, a text-guided weakly supervised framework that enhances MIL-based DFER by incorporating semantic guidance and coherent temporal modeling.
  arXiv  Detail & Related papers  (2025-11-14T04:49:58Z)
• FACE: Few-shot Adapter with Cross-view Fusion for Cross-subject EEG Emotion Recognition [57.08108545219043]
  Cross-subject EEG emotion recognition is challenged by significant inter-subject variability and intricately entangled intra-subject variability.<n>Recent few-shot learning paradigms attempt to address these limitations but often encounter catastrophic overfitting during subject-specific adaptation with limited samples.<n>This article introduces the few-shot adapter with a cross-view fusion method called FACE for cross-subject EEG emotion recognition.
  arXiv  Detail & Related papers  (2025-03-24T03:16:52Z)
• Controllable Talking Face Generation by Implicit Facial Keypoints Editing [6.036277153327655]
  We present ControlTalk, a talking face generation method to control face expression deformation based on driven audio. Our experiments show that our method is superior to state-of-the-art performance on widely used benchmarks, including HDTF and MEAD.
  arXiv  Detail & Related papers  (2024-06-05T02:54:46Z)
• CIAO! A Contrastive Adaptation Mechanism for Non-Universal Facial Expression Recognition [80.07590100872548]
  We propose Contrastive Inhibitory Adaptati On (CIAO), a mechanism that adapts the last layer of facial encoders to depict specific affective characteristics on different datasets. CIAO presents an improvement in facial expression recognition performance over six different datasets with very unique affective representations.
  arXiv  Detail & Related papers  (2022-08-10T15:46:05Z)
• Emotion Separation and Recognition from a Facial Expression by Generating the Poker Face with Vision Transformers [57.1091606948826]
  We propose a novel FER model, named Poker Face Vision Transformer or PF-ViT, to address these challenges. PF-ViT aims to separate and recognize the disturbance-agnostic emotion from a static facial image via generating its corresponding poker face. PF-ViT utilizes vanilla Vision Transformers, and its components are pre-trained as Masked Autoencoders on a large facial expression dataset.
  arXiv  Detail & Related papers  (2022-07-22T13:39:06Z)
• I Only Have Eyes for You: The Impact of Masks On Convolutional-Based Facial Expression Recognition [78.07239208222599]
  We evaluate how the recently proposed FaceChannel adapts towards recognizing facial expressions from persons with masks. We also perform specific feature-level visualization to demonstrate how the inherent capabilities of the FaceChannel to learn and combine facial features change when in a constrained social interaction scenario.
  arXiv  Detail & Related papers  (2021-04-16T20:03:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.