A Two-Stage Dual-Modality Model for Facial Emotional Expression Recognition

• URL: http://arxiv.org/abs/2603.12221v1
• Date: Thu, 12 Mar 2026 17:45:12 GMT
• Title: A Two-Stage Dual-Modality Model for Facial Emotional Expression Recognition
• Abstract summary: This paper addresses the expression (EXPR) recognition challenge in the 10th Affective Behavior Analysis in-the-Wild (ABAW) workshop and competition.<n>It requires frame-level classification of eight facial emotional expressions from unconstrained videos.<n>We propose a two-stage dual-modal (audio-visual) model to address these difficulties.
• Score: 5.014305646574725
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper addresses the expression (EXPR) recognition challenge in the 10th Affective Behavior Analysis in-the-Wild (ABAW) workshop and competition, which requires frame-level classification of eight facial emotional expressions from unconstrained videos. This task is challenging due to inaccurate face localization, large pose and scale variations, motion blur, temporal instability, and other confounding factors across adjacent frames. We propose a two-stage dual-modal (audio-visual) model to address these difficulties. Stage I focuses on robust visual feature extraction with a pretrained DINOv2-based encoder. Specifically, DINOv2 ViT-L/14 is used as the backbone, a padding-aware augmentation (PadAug) strategy is employed for image padding and data preprocessing from raw videos, and a mixture-of-experts (MoE) training head is introduced to enhance classifier diversity. Stage II addresses modality fusion and temporal consistency. For the visual modality, faces are re-cropped from raw videos at multiple scales, and the extracted visual features are averaged to form a robust frame-level representation. Concurrently, frame-aligned Wav2Vec 2.0 audio features are derived from short audio windows to provide complementary acoustic cues. These dual-modal features are integrated via a lightweight gated fusion module, followed by inference-time temporal smoothing. Experiments on the ABAW dataset demonstrate the effectiveness of the proposed method. The two-stage model achieves a Macro-F1 score of 0.5368 on the official validation set and 0.5122 +/- 0.0277 under 5-fold cross-validation, outperforming the official baselines.

Related papers

• Visual Representation Matters: Exploiting Temporal Differences in Video-to-Audio Generation [53.47090840868696]
  Video-to-audio (V2A) generation extends image-to-audio generation (I2A)<n>We introduce TD-V2A, which leverages temporal differences (TD) as the key representation that distinguishes V2A from I2A.<n>We show that effectively exploiting TD through our proposed framework significantly improves end-to-end V2A generation quality.
  arXiv  Detail & Related papers  (2026-08-05T14:28:05Z)
• Talker-T2AV: Joint Talking Audio-Video Generation with Autoregressive Diffusion Modeling [59.97930201616015]
  We propose Talker-T2AV, an autoregressive diffusion framework where high-level cross-modal modeling occurs in a shared backbone.<n>A shared autoregressive language model jointly reasons over audio and video in a unified patch-level token space.<n> Experiments on talking portrait benchmarks show Talker-T2AV outperforms dual-branch baselines in lip-sync accuracy, video quality, and audio quality.
  arXiv  Detail & Related papers  (2026-04-26T07:48:47Z)
• Team RAS in 10th ABAW Competition: Multimodal Valence and Arousal Estimation Approach [53.64487844936037]
  Our method combines three complementary modalities: face, behavior, and audio.<n>Experiments demonstrate that the proposed multimodal fusion strategy achieves a Concordance Correlation Coefficient (CCC) of 0.658 on the Aff-Wild2 development set.
  arXiv  Detail & Related papers  (2026-03-13T15:06:14Z)
• Multimodal Emotion Recognition via Bi-directional Cross-Attention and Temporal Modeling [2.8037951156321377]
  We propose a multimodal emotion recognition framework for the Expression (EXPR) Recognition task in the 10th Affective Behavior Analysis in-the-wild (ABAW) Challenge.<n>To model temporal dependencies in facial expression sequences, we employ a Temporal Convolutional Network (TCN) over fixed-length video windows.<n>In addition, we introduce a bi-directional cross-attention fusion module, in which visual and audio features interact symmetrically to enhance cross-modal contextualization.
  arXiv  Detail & Related papers  (2026-03-12T14:20:29Z)
• Diff-V2M: A Hierarchical Conditional Diffusion Model with Explicit Rhythmic Modeling for Video-to-Music Generation [26.273309051211204]
  Video-to-music (V2M) generation aims to create music that aligns with visual content.<n>We propose Diff-V2M, a general V2M framework based on a hierarchical conditional diffusion model.<n>For rhythm modeling, we begin by evaluating several rhythmic representations, including low-resolution mel-spectrograms, tempograms, and onset detection functions (ODF)
  arXiv  Detail & Related papers  (2025-11-12T08:02:06Z)
• StableDub: Taming Diffusion Prior for Generalized and Efficient Visual Dubbing [63.72095377128904]
  The visual dubbing task aims to generate mouth movements synchronized with the driving audio.<n>Audio-only driving paradigms inadequately capture speaker-specific lip habits.<n>Blind-inpainting approaches produce visual artifacts when handling obstructions.
  arXiv  Detail & Related papers  (2025-09-26T05:23:31Z)
• KSDiff: Keyframe-Augmented Speech-Aware Dual-Path Diffusion for Facial Animation [4.952724424448834]
  KSDiff is a Keyframe-Augmented Speech-Aware Dual-Path Diffusion framework.<n>It disentangles expression-related and head-pose-related features, while an autoregressive Keyframe Establishment Learning module predicts the most salient motion frames.<n>Experiments on HDTF and VoxCeleb demonstrate that KSDiff state-of-the-art performance, with improvements in both lip synchronization accuracy and head-pose naturalness.
  arXiv  Detail & Related papers  (2025-09-24T13:54:52Z)
• DreamVVT: Mastering Realistic Video Virtual Try-On in the Wild via a Stage-Wise Diffusion Transformer Framework [26.661935208583756]
  virtual try-on (VVT) technology has garnered considerable academic interest owing to its promising applications in e-commerce advertising and entertainment.<n>We propose DreamVVT, which is inherently capable of leveraging diverse unpaired human-centric data to enhance adaptability in real-world scenarios.<n>In the first stage, we sample representative frames from the input video and utilize a multi-frame try-on model integrated with a vision-language model (VLM), to synthesize high-fidelity and semantically consistent try-on images.<n>In the second stage, skeleton maps together with fine-grained motion and appearance descriptions are
  arXiv  Detail & Related papers  (2025-08-04T18:27:55Z)
• TAViS: Text-bridged Audio-Visual Segmentation with Foundation Models [123.17643568298116]
  We present TAViS, a novel framework that textbfcouples the knowledge of multimodal foundation models for cross-modal alignment.<n> effectively combining these models poses two key challenges: the difficulty in transferring the knowledge between SAM2 and ImageBind due to their different feature spaces, and the insufficiency of using only segmentation loss for supervision.<n>Our approach achieves superior performance on single-source, multi-source, semantic datasets, and excels in zero-shot settings.
  arXiv  Detail & Related papers  (2025-06-13T03:19:47Z)
• Harnessing Diffusion Models for Visual Perception with Meta Prompts [68.78938846041767]
  We propose a simple yet effective scheme to harness a diffusion model for visual perception tasks. We introduce learnable embeddings (meta prompts) to the pre-trained diffusion models to extract proper features for perception. Our approach achieves new performance records in depth estimation tasks on NYU depth V2 and KITTI, and in semantic segmentation task on CityScapes.
  arXiv  Detail & Related papers  (2023-12-22T14:40:55Z)
• From Static to Dynamic: Adapting Landmark-Aware Image Models for Facial Expression Recognition in Videos [88.08209394979178]
  Dynamic facial expression recognition (DFER) in the wild is still hindered by data limitations. We introduce a novel Static-to-Dynamic model (S2D) that leverages existing SFER knowledge and dynamic information implicitly encoded in extracted facial landmark-aware features.
  arXiv  Detail & Related papers  (2023-12-09T03:16:09Z)
• Temporal Distinct Representation Learning for Action Recognition [139.93983070642412]
  Two-Dimensional Convolutional Neural Network (2D CNN) is used to characterize videos. Different frames of a video share the same 2D CNN kernels, which may result in repeated and redundant information utilization. We propose a sequential channel filtering mechanism to excite the discriminative channels of features from different frames step by step, and thus avoid repeated information extraction. Our method is evaluated on benchmark temporal reasoning datasets Something-Something V1 and V2, and it achieves visible improvements over the best competitor by 2.4% and 1.3%, respectively.
  arXiv  Detail & Related papers  (2020-07-15T11:30:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.