Multi-Track Multimodal Learning on iMiGUE: Micro-Gesture and Emotion Recognition

• URL: http://arxiv.org/abs/2512.23291v1
• Date: Mon, 29 Dec 2025 08:22:46 GMT
• Title: Multi-Track Multimodal Learning on iMiGUE: Micro-Gesture and Emotion Recognition
• Authors: Arman Martirosyan, Shahane Tigranyan, Maria Razzhivina, Artak Aslanyan, Nazgul Salikhova, Ilya Makarov, Andrey Savchenko, Aram Avetisyan,
• Abstract summary: We present two frameworks designed to tackle both problems on the iMiGUE dataset.<n>For micro-gesture classification, we explore complementary strengths of RGB and 3D pose-based representations.<n>For emotion recognition, our framework extends to behavior-based emotion prediction.
• Score: 4.909448578374012
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Micro-gesture recognition and behavior-based emotion prediction are both highly challenging tasks that require modeling subtle, fine-grained human behaviors, primarily leveraging video and skeletal pose data. In this work, we present two multimodal frameworks designed to tackle both problems on the iMiGUE dataset. For micro-gesture classification, we explore the complementary strengths of RGB and 3D pose-based representations to capture nuanced spatio-temporal patterns. To comprehensively represent gestures, video, and skeletal embeddings are extracted using MViTv2-S and 2s-AGCN, respectively. Then, they are integrated through a Cross-Modal Token Fusion module to combine spatial and pose information. For emotion recognition, our framework extends to behavior-based emotion prediction, a binary classification task identifying emotional states based on visual cues. We leverage facial and contextual embeddings extracted using SwinFace and MViTv2-S models and fuse them through an InterFusion module designed to capture emotional expressions and body gestures. Experiments conducted on the iMiGUE dataset, within the scope of the MiGA 2025 Challenge, demonstrate the robust performance and accuracy of our method in the behavior-based emotion prediction task, where our approach secured 2nd place.

Related papers

• DEFT-LLM: Disentangled Expert Feature Tuning for Micro-Expression Recognition [16.903294278064667]
  We propose DEFT-LLM, which achieves semantic alignment by multi-expert disentanglement.<n>We first introduce Uni-MER, a motion-driven instruction designed to align text with local facial motion.<n>We then design an architecture with three experts to decouple facial dynamics into independent representations.
  arXiv  Detail & Related papers  (2025-11-14T04:21:24Z)
• Hybrid-supervised Hypergraph-enhanced Transformer for Micro-gesture Based Emotion Recognition [30.016692048849226]
  Micro-gestures are unconsciously performed body gestures that can convey the emotion states of humans.<n>We propose to recognize the emotion states based on the micro-gestures by reconstructing the behavior patterns with a hypergraph-enhanced Transformer.<n>The proposed method is evaluated on two publicly available datasets, namely iMiGUE and SMG.
  arXiv  Detail & Related papers  (2025-07-20T08:27:56Z)
• Milmer: a Framework for Multiple Instance Learning based Multimodal Emotion Recognition [16.616341358877243]
  This study addresses the challenges of emotion recognition by integrating facial expression analysis with electroencephalogram (EEG) signals.<n>The proposed framework employs a transformer-based fusion approach to effectively integrate visual and physiological modalities.<n>A key innovation of this work is the adoption of a multiple instance learning (MIL) approach, which extracts meaningful information from multiple facial expression images.
  arXiv  Detail & Related papers  (2025-02-01T20:32:57Z)
• Through-The-Mask: Mask-based Motion Trajectories for Image-to-Video Generation [52.337472185022136]
  We consider the task of Image-to-Video (I2V) generation, which involves transforming static images into realistic video sequences based on a textual description.<n>We propose a two-stage compositional framework that decomposes I2V generation into: (i) An explicit intermediate representation generation stage, followed by (ii) A video generation stage that is conditioned on this representation.<n>We evaluate our method on challenging benchmarks with multi-object and high-motion scenarios and empirically demonstrate that the proposed method achieves state-of-the-art consistency.
  arXiv  Detail & Related papers  (2025-01-06T14:49:26Z)
• MEMO-Bench: A Multiple Benchmark for Text-to-Image and Multimodal Large Language Models on Human Emotion Analysis [53.012111671763776]
  This study introduces MEMO-Bench, a comprehensive benchmark consisting of 7,145 portraits, each depicting one of six different emotions. Results demonstrate that existing T2I models are more effective at generating positive emotions than negative ones. Although MLLMs show a certain degree of effectiveness in distinguishing and recognizing human emotions, they fall short of human-level accuracy.
  arXiv  Detail & Related papers  (2024-11-18T02:09:48Z)
• Micro-Expression Recognition by Motion Feature Extraction based on Pre-training [6.015288149235598]
  We propose a novel motion extraction strategy (MoExt) for the micro-expression recognition task. In MoExt, shape features and texture features are first extracted separately from onset and apex frames, and then motion features related to MEs are extracted based on shape features of both frames. The effectiveness of proposed method is validated on three commonly used datasets.
  arXiv  Detail & Related papers  (2024-07-10T03:51:34Z)
• Two in One Go: Single-stage Emotion Recognition with Decoupled Subject-context Transformer [78.35816158511523]
  We present a single-stage emotion recognition approach, employing a Decoupled Subject-Context Transformer (DSCT) for simultaneous subject localization and emotion classification. We evaluate our single-stage framework on two widely used context-aware emotion recognition datasets, CAER-S and EMOTIC.
  arXiv  Detail & Related papers  (2024-04-26T07:30:32Z)
• AIMDiT: Modality Augmentation and Interaction via Multimodal Dimension Transformation for Emotion Recognition in Conversations [57.99479708224221]
  We propose a novel framework called AIMDiT to solve the problem of multimodal fusion of deep features. Experiments conducted using our AIMDiT framework on the public benchmark dataset MELD reveal 2.34% and 2.87% improvements in terms of the Acc-7 and w-F1 metrics.
  arXiv  Detail & Related papers  (2024-04-12T11:31:18Z)
• Joint Multimodal Transformer for Emotion Recognition in the Wild [49.735299182004404]
  Multimodal emotion recognition (MMER) systems typically outperform unimodal systems. This paper proposes an MMER method that relies on a joint multimodal transformer (JMT) for fusion with key-based cross-attention.
  arXiv  Detail & Related papers  (2024-03-15T17:23:38Z)
• Disentangled Interaction Representation for One-Stage Human-Object Interaction Detection [70.96299509159981]
  Human-Object Interaction (HOI) detection is a core task for human-centric image understanding. Recent one-stage methods adopt a transformer decoder to collect image-wide cues that are useful for interaction prediction. Traditional two-stage methods benefit significantly from their ability to compose interaction features in a disentangled and explainable manner.
  arXiv  Detail & Related papers  (2023-12-04T08:02:59Z)
• iMiGUE: An Identity-free Video Dataset for Micro-Gesture Understanding and Emotion Analysis [23.261770969903065]
  iMiGUE is identity-free video dataset for Micro-Gesture Understanding and Emotion analysis (iMiGUE) iMiGUE focuses on micro-gesture, i.e., unintentional behaviors driven by inner feelings.
  arXiv  Detail & Related papers  (2021-07-01T08:15:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.