Personalized Cross-Modal Emotional Correlation Learning for Speech-Preserving Facial Expression Manipulation

• URL: http://arxiv.org/abs/2604.25255v1
• Date: Tue, 28 Apr 2026 06:02:23 GMT
• Title: Personalized Cross-Modal Emotional Correlation Learning for Speech-Preserving Facial Expression Manipulation
• Abstract summary: Speech-preserving facial expression manipulation aims to enhance human expressiveness without altering mouth movements tied to the original speech.<n>Current Visual-Language Models (VLMs) can extract aligned visual and semantic features, making them a promising source of supervision.<n>We propose a Personalized Cross-Modal Emotional Correlation Learning algorithm that refines VLM-based supervision.
• Score: 63.90351637818903
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Speech-preserving facial expression manipulation (SPFEM) aims to enhance human expressiveness without altering mouth movements tied to the original speech. A primary challenge in this domain is the scarcity of paired data, namely aligned frames of the same individual with identical speech but different expressions, which impedes direct supervision for emotional manipulation. While current Visual-Language Models (VLMs) can extract aligned visual and semantic features, making them a promising source of supervision, their direct application is limited. To this end, we propose a Personalized Cross-Modal Emotional Correlation Learning (PCMECL) algorithm that refines VLM-based supervision through two major improvements. First, standard VLMs rely on a single generic prompt for each emotion, failing to capture expressive variations among individuals. PCMECL addresses this limitation by conditioning on individual visual information to learn personalized prompts, thereby establishing more fine-grained visual-semantic correlations. Second, even with personalization, inherent discrepancies persist between the visual and semantic feature distributions. To bridge this modality gap, PCMECL employs feature differencing to correlate the modalities, providing more precisely aligned supervision by matching the change in visual features to the change in semantic features. As a plug-and-play module, PCMECL can be seamlessly integrated into existing SPFEM models. Extensive experiments across various datasets demonstrate the superior efficacy of our algorithm.

Related papers

• Let Me Look at You: Advanced Facial Expression Modeling for Conversational Speech Synthesis [36.79172273160707]
  Facial expressions encode subtle and rich affective cues that are crucial for empathetic speech interaction.<n>We propose FacialTalker, a facial-expression-aware CSS framework built upon a large language model backbone.<n>We construct VSDD-1K, a large-scale multimodal dialogue dataset collected through a fully automated pipeline from real-world Internet conversations.
  arXiv  Detail & Related papers  (2026-07-27T13:42:37Z)
• Adaptive Physical-Facial Representation Fusion via Subject-Invariant Cross-Modal Prompt Tuning for Video-Based Emotion Recognition [14.850489141041017]
  We propose a cross-modal cross-tuning prompt-tuning method for video-based emotion recognition.<n>The proposed method consistently outperforms strong baselines in both recognition accuracy and generalization ability.
  arXiv  Detail & Related papers  (2026-05-07T05:33:31Z)
• Learning Spatial-Temporal Coherent Correlations for Speech-Preserving Facial Expression Manipulation [75.05465510260944]
  Speech-preserving facial expression manipulation (SPFEM) aims to modify facial emotions while meticulously maintaining the mouth animation associated with spoken content.<n>Current works depend on inaccessible paired training samples for the person, where two aligned frames exhibit the same speech content yet differ in emotional expression, limiting the SPFEM applications in real-world scenarios.<n>We propose a novel spatial-temporal coherent correlation learning (STCCL) algorithm, which models the aforementioned correlations as explicit metrics and integrates the metrics to supervise manipulating facial expression and meanwhile better preserving the facial animation of spoken content.
  arXiv  Detail & Related papers  (2026-04-22T06:20:10Z)
• Bidirectional Learning of Facial Action Units and Expressions via Structured Semantic Mapping across Heterogeneous Datasets [85.74213192818668]
  We propose a Structured Semantic Mapping (SSM) framework for bidirectional AU--FE learning under different data domains and heterogeneous supervision.<n>SSM consists of three key components: (1) a shared visual backbone that learns unified facial representations from dynamic AU and FE videos; (2) semantic mediation via a Textual Semantic Prototype (TSP) module; and (3) a Dynamic Prior Mapping (DPM) module that incorporates prior knowledge derived from the Facial Action Coding System.
  arXiv  Detail & Related papers  (2026-04-12T09:08:32Z)
• Multimodal Emotion Recognition via Bi-directional Cross-Attention and Temporal Modeling [2.8037951156321377]
  We propose a multimodal emotion recognition framework for the Expression task in the 10th Affective Behavior Analysis in-the-wild (ABAW) Challenge.<n>Our framework builds on large-scale pre-trained models for visual and audio representation learning and integrates them in a unified multimodal architecture.<n> Experimental results on the ABAW 10th EXPR benchmark demonstrate the effectiveness of the proposed framework.
  arXiv  Detail & Related papers  (2026-03-12T14:20:29Z)
• Text-guided Weakly Supervised Framework for Dynamic Facial Expression Recognition [49.41688891301643]
  Dynamic facial expression recognition aims to identify emotional states by modeling the temporal changes in facial movements across video sequences.<n>A key challenge in DFER is the many-to-one labeling problem, where a video composed of numerous frames is assigned a single emotion label.<n>We propose TG-DFER, a text-guided weakly supervised framework that enhances MIL-based DFER by incorporating semantic guidance and coherent temporal modeling.
  arXiv  Detail & Related papers  (2025-11-14T04:49:58Z)
• Multimodal Prompt Alignment for Facial Expression Recognition [24.470095812039286]
  MPA-FER provides fine-grained semantic guidance to the learning process of prompted visual features.<n>Our framework outperforms state-of-the-art methods on three FER benchmark datasets.
  arXiv  Detail & Related papers  (2025-06-26T05:28:57Z)
• VAEmo: Efficient Representation Learning for Visual-Audio Emotion with Knowledge Injection [50.57849622045192]
  We propose VAEmo, an efficient framework for emotion-centric joint VA representation learning with external knowledge injection.<n>VAEmo achieves state-of-the-art performance with a compact design, highlighting the benefit of unified cross-modal encoding and emotion-aware semantic guidance.
  arXiv  Detail & Related papers  (2025-05-05T03:00:51Z)
• AMuSE: Adaptive Multimodal Analysis for Speaker Emotion Recognition in Group Conversations [39.79734528362605]
  Multimodal Attention Network captures cross-modal interactions at various levels of spatial abstraction. AMuSE model condenses both spatial and temporal features into two dense descriptors: speaker-level and utterance-level.
  arXiv  Detail & Related papers  (2024-01-26T19:17:05Z)
• CLIPER: A Unified Vision-Language Framework for In-the-Wild Facial Expression Recognition [1.8604727699812171]
  We propose a unified framework for both static and dynamic facial Expression Recognition based on CLIP. We introduce multiple expression text descriptors (METD) to learn fine-grained expression representations that make CLIPER more interpretable.
  arXiv  Detail & Related papers  (2023-03-01T02:59:55Z)
• MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretraining [138.86293836634323]
  MaskCLIP incorporates a newly proposed masked self-distillation into contrastive language-image pretraining. MaskCLIP achieves superior results in linear probing, finetuning, and zero-shot performance with the guidance of the language encoder.
  arXiv  Detail & Related papers  (2022-08-25T17:59:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.