A Multi-Modal Framework with Cross-Subject Pseudo-Labeling and Semantic Alignment for Micro-Gesture Recognition

• URL: http://arxiv.org/abs/2606.13030v1
• Date: Thu, 11 Jun 2026 08:04:42 GMT
• Title: A Multi-Modal Framework with Cross-Subject Pseudo-Labeling and Semantic Alignment for Micro-Gesture Recognition
• Abstract summary: Micro-gestures (MGs) are spontaneous and subtle body movements that frequently convey hidden human emotions.<n> Recognizing MGs in untrimmed videos remains highly challenging due to their extremely low signal-to-noise ratio.<n>We propose a comprehensive multi-modal framework for Track 1 of the 4th MiGA-IJCAI Challenge.
• Score: 21.577609169548168
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Micro-gestures (MGs) are spontaneous and subtle body movements that frequently convey hidden human emotions. Recognizing MGs in untrimmed videos remains highly challenging due to their extremely low signal-to-noise ratio, severe long-tailed class distribution, and the inherent domain shift encountered in cross-subject evaluation scenarios. In this paper, we propose a comprehensive multi-modal framework for Track 1 of the 4th MiGA-IJCAI Challenge. To capture fine-grained representations, we design a saliency-guided multi-modal extraction pipeline integrating 68-keypoint skeleton joint coordinates, 3D heatmap volumes, and high-resolution RGB visual features. We introduce a gentle square-root smoothed weighting mechanism paired with an Orthogonal Semantic Embedding Loss to protect tail classes without compromising overall recognition capabilities. More importantly, to bridge the cross-subject generalization gap, we propose a Cross-Modal Pseudo-Labeling (CMPL) strategy for unsupervised domain adaptation, which significantly boosts single-modal robustness. A temperature-scaled soft-voting mechanism is finally utilized to alleviate overconfidence during late fusion. Extensive experiments demonstrate that our framework achieves a competitive F1-score of 68.13\%, securing the 4th place.

Related papers

• Promptable Concept Segmentation from Above: Evaluating SAM 3's Zero-Shot and One-Shot Capabilities in Remote Sensing [0.42970700836450487]
  We present a comprehensive, multi-task empirical evaluation across remote sensing scene classification, object detection, and instance segmentation.<n>Our results demonstrate that SAM 3 avoids the overfitting commonly seen in legacy domain-adapted models.<n>It remains fundamentally constrained by sub-pixel resolution limits and overhead semantic blind spots.
  arXiv  Detail & Related papers  (2026-07-10T16:33:05Z)
• Motion Reinforces Appearance: RGB-Skeleton Gated Residual Fusion for Micro-Gesture Online Recognition [18.736764781779208]
  We propose DyFADet+, which extends DyFADet into a dual-stream RGB-skeleton framework.<n>On the SMG dataset, our method achieves an F1 score of 40.88, ranking 2nd in the Micro-gesture Online Recognition track.
  arXiv  Detail & Related papers  (2026-06-10T04:12:36Z)
• Hierarchical Consistency Learning for Test-time Adaptation in Camouflage Perception [50.278200968044665]
  Camouflaged object detection (COD) aims to localize targets that exhibit minimal perceptual differences from backgrounds through physical attributes.<n>Existing methods, constrained by the static train-then-freeze paradigm, suffer from domain rigidity and dependency annotation.<n>We propose the hierarchical consistency learning framework, which integrates test-time adaptation for dynamic representation recalibration.
  arXiv  Detail & Related papers  (2026-05-25T09:57:46Z)
• Gait Recognition via Deep Residual Networks and Multi-Branch Feature Fusion [5.585970057888865]
  We present a high-precision gait recognition framework that deeply extracts and fuses gait dynamics with body shape characteristics.<n>A 50-layer Residual Network is leveraged within a deep feature extraction module to capture hierarchically rich and discriminative representations.<n>Experiments on the cross-view multi-condition CASIA-B benchmark demonstrate that our method achieves a Rank-1 accuracy of 94.52% under normal walking.
  arXiv  Detail & Related papers  (2026-04-30T03:09:51Z)
• Image Classification via Random Dilated Convolution with Multi-Branch Feature Extraction and Context Excitation [3.9240527145303123]
  RDCNet is a novel architecture built upon ResNet-34 that integrates three synergistic innovations to address these limitations.<n>RDCNet consistently achieves state-of-the-art classification accuracy, outperforming the second-best competing methods by margins of 0.02%, 1.12%, 0.18%, 4.73%, respectively.
  arXiv  Detail & Related papers  (2026-04-28T03:49:35Z)
• Bidirectional Cross-Modal Prompting for Event-Frame Asymmetric Stereo [54.58078274676216]
  Event cameras offer an alternative visual representation with higher dynamic range free of such limitations.<n>In this paper, we introduce Bi-CMPStereo, a novel bidirectional cross-modal prompting framework.<n>Our approach learns finely aligned stereo representations within a target canonical space and integrates complementary representations by projecting each modality into both event and frame domains.
  arXiv  Detail & Related papers  (2026-04-16T17:59:58Z)
• Complementarity-Supervised Spectral-Band Routing for Multimodal Emotion Recognition [60.20529806857076]
  Multimodal emotion recognition fuses cues such as text, video, and audio to understand individual emotional states.<n>Prior methods face two main limitations: mechanically relying on independent unimodal performance, and coarse-grained fusion conflicting with the fine-grained representations required by emotion tasks.<n>We propose the Complementarity-Supervised Multi-Band Expert Network, named Atsuko, to model fine-grained complementary features via multi-scale band decomposition and expert collaboration.
  arXiv  Detail & Related papers  (2026-03-07T03:58:48Z)
• BHaRNet: Reliability-Aware Body-Hand Modality Expertized Networks for Fine-grained Skeleton Action Recognition [12.973657570368317]
  This work presents a probabilistic dual-stream framework that unifies reliability modeling and multi-modal integration, generalizing expertized learning under uncertainty across both intra-skeleton and cross-modal domains.<n>The framework comprises three key components: (1) a calibration-free preprocessing pipeline that removes canonical-space transformations and learns directly from native coordinates; (2) a probabilistic Noisy-OR fusion that stabilizes reliability-aware dual-stream learning without requiring explicit confidence supervision; and (3) an intra- to cross-modal ensemble that couples four skeleton modalities (Joint, Bone, Joint Motion, and Bone Motion) to RGB representations, bridging structural and visual motion
  arXiv  Detail & Related papers  (2026-01-01T15:13:11Z)
• X-ReID: Multi-granularity Information Interaction for Video-Based Visible-Infrared Person Re-Identification [79.37768038337971]
  We propose a novel cross-modality feature learning framework named X-ReID for VVI-ReID.<n> Specifically, we first propose a Cross-modality Prototype Collaboration (CPC)<n>Then, a Multi-granularity Information Interaction (MII) is designed, incorporating short-term interactions from adjacent frames, long-term cross-frame information fusion, and cross-modality feature alignment.
  arXiv  Detail & Related papers  (2025-11-22T07:57:15Z)
• Adversarial Robustness for Unified Multi-Modal Encoders via Efficient Calibration [12.763688592842717]
  We present the first comprehensive study of adversarial vulnerability in unified multi-modal encoders.<n>Non-visual inputs, such as audio and point clouds, are especially fragile.<n>Our method improves adversarial robustness by up to 47.3 percent at epsilon = 4/255.
  arXiv  Detail & Related papers  (2025-05-17T08:26:04Z)
• Unified Frequency-Assisted Transformer Framework for Detecting and Grounding Multi-Modal Manipulation [109.1912721224697]
  We present the Unified Frequency-Assisted transFormer framework, named UFAFormer, to address the DGM4 problem. By leveraging the discrete wavelet transform, we decompose images into several frequency sub-bands, capturing rich face forgery artifacts. Our proposed frequency encoder, incorporating intra-band and inter-band self-attentions, explicitly aggregates forgery features within and across diverse sub-bands.
  arXiv  Detail & Related papers  (2023-09-18T11:06:42Z)
• Cross-Attention is Not Enough: Incongruity-Aware Dynamic Hierarchical Fusion for Multimodal Affect Recognition [69.32305810128994]
  Incongruity between modalities poses a challenge for multimodal fusion, especially in affect recognition. We propose the Hierarchical Crossmodal Transformer with Dynamic Modality Gating (HCT-DMG), a lightweight incongruity-aware model. HCT-DMG: 1) outperforms previous multimodal models with a reduced size of approximately 0.8M parameters; 2) recognizes hard samples where incongruity makes affect recognition difficult; 3) mitigates the incongruity at the latent level in crossmodal attention.
  arXiv  Detail & Related papers  (2023-05-23T01:24:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.