SAC$^2$-Net: Semantic Anchoring and Complementary-Consensus Fusion for Multimodal Micro-Expression Recognition

• URL: http://arxiv.org/abs/2606.25542v1
• Date: Wed, 24 Jun 2026 08:20:03 GMT
• Title: SAC$^2$-Net: Semantic Anchoring and Complementary-Consensus Fusion for Multimodal Micro-Expression Recognition
• Abstract summary: Micro-expression recognition (MER) is challenging due to subtle facial movements, limited data, and the ambiguous relationship between Action Units (AUs) and emotion categories.<n>We propose SAC$2$-Net, a Semantic Anchoring and Complementary-Consensus Network for multimodal MER.<n>We show that SAC$2$-Net achieves state-of-the-art or highly competitive performance across coarse-grained, fine-grained, large-scale, and cross-dataset evaluation settings.
• Score: 5.567888983005847
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Micro-expression recognition (MER) is challenging due to subtle facial movements, limited data, and the ambiguous relationship between Action Units (AUs) and emotion categories. Optical flow and motion magnification have been widely used to describe subtle facial dynamics from different perspectives: the former captures local motion displacement, while the latter amplifies weak appearance changes. In this work, we observe that these two modalities often exhibit asymmetric failure patterns: one modality may become noisy, distorted, or uninformative, while the other still preserves discriminative AU-related evidence. This phenomenon reveals their complementarity, but also raises two key challenges for fusion: cross-modal heterogeneity and spatially varying modality reliability. Motivated by this observation, we propose SAC$^2$-Net, a Semantic Anchoring and Complementary-Consensus Network for multimodal MER, which first aligns visual modalities with semantic anchors and then performs reliability-aware fusion. To reduce cross-modal heterogeneity before fusion, we introduce Semantic Anchoring Soft Alignment (SASA), which converts activated AUs into textual prompts and uses them as stable semantic anchors to align motion-magnified and optical-flow representations. Unlike hard contrastive learning, SASA constructs hierarchical AU-aware soft labels to preserve semantic proximity among samples with overlapping or anatomically related AU patterns. Based on the aligned representations, Complementary-Consensus Fusion (CCF) first repairs unreliable local evidence through complementary exchange and then enforces a shared spatial focus through consensus refinement. Extensive experiments on five MER benchmarks show that SAC$^2$-Net achieves state-of-the-art or highly competitive performance across coarse-grained, fine-grained, large-scale, and cross-dataset evaluation settings.

Related papers

• UMSS: Towards Unsupervised Multi-modal Semantic Segmentation [6.785303759588842]
  We make the first attempt to address the problem of Unsupervised Multimodal Semantic (UMSS)<n>We propose UniM2, a novel framework built on DINOv3 that transforms conventional fusion methods into consistent performance gains.<n>Our key idea is to learn a unified latent space driven by Cross Modal Correspondence Synergy (CMCS) to extract intrinsic shared semantic cues.
  arXiv  Detail & Related papers  (2026-07-14T05:44:29Z)
• STAR: Semantic-Temporal Adaptive Representation Learning for Few-Shot Action Recognition [23.546777614096424]
  Few-shot action recognition (FSAR) requires models to generalize to novel action categories from only a handful of annotated samples.<n>Despite progress with vision-language models, existing approaches still suffer from semantic-temporal misalignment.<n>We propose Semantic Temporal Adaptive Representation Learning (STAR), a unified framework consisting of a semantic-alignment component and a temporal-aware component.
  arXiv  Detail & Related papers  (2026-05-13T08:54:38Z)
• Chain of Modality: From Static Fusion to Dynamic Orchestration in Omni-MLLMs [84.3271821505699]
  Chain of Modality (CoM) is an agentic framework that transitions multimodal fusion from passive concatenation to dynamic orchestration.<n> operating in either a training-free or a data-efficient SFT setting, CoM achieves robust and consistent generalization across diverse benchmarks.
  arXiv  Detail & Related papers  (2026-04-16T01:21:14Z)
• Bridging Pixels and Words: Mask-Aware Local Semantic Fusion for Multimodal Media Verification [13.571218577944032]
  MaLSF (Mask-aware Local Semantic Fusion) is a novel framework that shifts the paradigm to active, bidirectional verification.<n>It uses mask-label pairs as semantic anchors to bridge pixels and words.<n>MaLSF achieves state-of-the-art performance on both the DGM4 and multimodal fake news detection tasks.
  arXiv  Detail & Related papers  (2026-03-27T03:38:38Z)
• Explaining multimodal LLMs via intra-modal token interactions [55.27436637894534]
  Multimodal Large Language Models (MLLMs) have achieved remarkable success across diverse vision-language tasks, yet their internal decision-making mechanisms remain insufficiently understood.<n>We propose enhancing interpretability by leveraging intra-modal interaction.
  arXiv  Detail & Related papers  (2025-09-26T14:39:13Z)
• Rethinking Occlusion in FER: A Semantic-Aware Perspective and Go Beyond [10.015531203047598]
  We present ORSANet, which introduces auxiliary multi-modal semantic guidance to disambiguate facial occlusion.<n>We also introduce facial landmarks as sparse geometric prior to mitigate intrinsic noises in FER, such as identity and gender biases.<n>Our proposed ORSANet achieves SOTA recognition performance.
  arXiv  Detail & Related papers  (2025-07-21T09:04:29Z)
• Spatial Semantic Recurrent Mining for Referring Image Segmentation [63.34997546393106]
  We propose Stextsuperscript2RM to achieve high-quality cross-modality fusion. It follows a working strategy of trilogy: distributing language feature, spatial semantic recurrent coparsing, and parsed-semantic balancing. Our proposed method performs favorably against other state-of-the-art algorithms.
  arXiv  Detail & Related papers  (2024-05-15T00:17:48Z)
• Exploring Homogeneous and Heterogeneous Consistent Label Associations for Unsupervised Visible-Infrared Person ReID [57.500045584556794]
  We introduce a Modality-Unified Label Transfer (MULT) module that simultaneously accounts for both homogeneous and heterogeneous fine-grained instance-level structures.<n>The proposed MULT ensures that the generated pseudo-labels maintain alignment across modalities while upholding structural consistency within intra-modality.<n> Experiments demonstrate that our proposed method outperforms existing state-of-the-art USL-VI-ReID methods.
  arXiv  Detail & Related papers  (2024-02-01T15:33:17Z)
• Object Segmentation by Mining Cross-Modal Semantics [68.88086621181628]
  We propose a novel approach by mining the Cross-Modal Semantics to guide the fusion and decoding of multimodal features. Specifically, we propose a novel network, termed XMSNet, consisting of (1) all-round attentive fusion (AF), (2) coarse-to-fine decoder (CFD), and (3) cross-layer self-supervision.
  arXiv  Detail & Related papers  (2023-05-17T14:30:11Z)
• Semantic-aligned Fusion Transformer for One-shot Object Detection [18.58772037047498]
  One-shot object detection aims at detecting novel objects according to merely one given instance. Current approaches explore various feature fusions to obtain directly transferable meta-knowledge. We propose a simple but effective architecture named Semantic-aligned Fusion Transformer (SaFT) to resolve these issues.
  arXiv  Detail & Related papers  (2022-03-17T05:38:47Z)
• Group Gated Fusion on Attention-based Bidirectional Alignment for Multimodal Emotion Recognition [63.07844685982738]
  This paper presents a new model named as Gated Bidirectional Alignment Network (GBAN), which consists of an attention-based bidirectional alignment network over LSTM hidden states. We empirically show that the attention-aligned representations outperform the last-hidden-states of LSTM significantly. The proposed GBAN model outperforms existing state-of-the-art multimodal approaches on the IEMOCAP dataset.
  arXiv  Detail & Related papers  (2022-01-17T09:46:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.