Hierarchical Vision-Language Interaction for Facial Action Unit Detection

• URL: http://arxiv.org/abs/2602.14425v1
• Date: Mon, 16 Feb 2026 03:22:05 GMT
• Title: Hierarchical Vision-Language Interaction for Facial Action Unit Detection
• Authors: Yong Li, Yi Ren, Yizhe Zhang, Wenhua Zhang, Tianyi Zhang, Muyun Jiang, Guo-Sen Xie, Cuntai Guan,
• Abstract summary: We propose a Hierarchical Vision-language Interaction for AU Understanding (HiVA) method to enhance AU detection.<n>HiVA employs a large language model to generate diverse and contextually rich AU descriptions to strengthen language-based representation learning.<n> experiments show that HiVA consistently surpasses state-of-the-art approaches.
• Score: 44.02409932746335
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Facial Action Unit (AU) detection seeks to recognize subtle facial muscle activations as defined by the Facial Action Coding System (FACS). A primary challenge w.r.t AU detection is the effective learning of discriminative and generalizable AU representations under conditions of limited annotated data. To address this, we propose a Hierarchical Vision-language Interaction for AU Understanding (HiVA) method, which leverages textual AU descriptions as semantic priors to guide and enhance AU detection. Specifically, HiVA employs a large language model to generate diverse and contextually rich AU descriptions to strengthen language-based representation learning. To capture both fine-grained and holistic vision-language associations, HiVA introduces an AU-aware dynamic graph module that facilitates the learning of AU-specific visual representations. These features are further integrated within a hierarchical cross-modal attention architecture comprising two complementary mechanisms: Disentangled Dual Cross-Attention (DDCA), which establishes fine-grained, AU-specific interactions between visual and textual features, and Contextual Dual Cross-Attention (CDCA), which models global inter-AU dependencies. This collaborative, cross-modal learning paradigm enables HiVA to leverage multi-grained vision-based AU features in conjunction with refined language-based AU details, culminating in robust and semantically enriched AU detection capabilities. Extensive experiments show that HiVA consistently surpasses state-of-the-art approaches. Besides, qualitative analyses reveal that HiVA produces semantically meaningful activation patterns, highlighting its efficacy in learning robust and interpretable cross-modal correspondences for comprehensive facial behavior analysis.

Related papers

• Contextualized Representation Learning for Effective Human-Object Interaction Detection [17.242400169885453]
  Human-Object Interaction (HOI) detection aims to simultaneously localize human-object pairs and recognize their interactions.<n>We introduce a Contextualized Representation Learning that integrates both affordance-guided reasoning and contextual prompts.<n>Our proposed method demonstrates superior performance on both the HICO-Det and V-COCO datasets in most scenarios.
  arXiv  Detail & Related papers  (2025-09-16T08:03:16Z)
• HiLa: Hierarchical Vision-Language Collaboration for Cancer Survival Prediction [55.00788339683146]
  We propose a novel Hierarchical vision-Language collaboration framework for improved survival prediction.<n> Specifically, HiLa employs pretrained feature extractors to generate hierarchical visual features from WSIs at both patch and region levels.<n>This ap-proach enables the comprehensive learning of discriminative visual features cor-responding to different survival-related attributes from prompts.
  arXiv  Detail & Related papers  (2025-07-07T02:06:25Z)
• Linguistics-aware Masked Image Modeling for Self-supervised Scene Text Recognition [50.86415025650168]
  Masked image modeling (MIM) tends to exploit local structures to reconstruct visual patterns, resulting in limited linguistic knowledge.<n>We propose a Linguistics-aware Masked Image Modeling (LMIM) approach, which channels the linguistic information into the decoding process of MIM through a separate branch.
  arXiv  Detail & Related papers  (2025-03-24T14:53:35Z)
• Towards End-to-End Explainable Facial Action Unit Recognition via Vision-Language Joint Learning [48.70249675019288]
  We propose an end-to-end Vision-Language joint learning network for explainable facial action units (AUs) recognition. The proposed approach achieves superior performance over the state-of-the-art methods on most metrics.
  arXiv  Detail & Related papers  (2024-08-01T15:35:44Z)
• Emotic Masked Autoencoder with Attention Fusion for Facial Expression Recognition [1.4374467687356276]
  This paper presents an innovative approach integrating the MAE-Face self-supervised learning (SSL) method and multi-view Fusion Attention mechanism for expression classification. We suggest easy-to-implement and no-training frameworks aimed at highlighting key facial features to determine if such features can serve as guides for the model. The efficacy of this method is validated by improvements in model performance on the Aff-wild2 dataset.
  arXiv  Detail & Related papers  (2024-03-19T16:21:47Z)
• Towards Zero-shot Human-Object Interaction Detection via Vision-Language Integration [14.678931157058363]
  We propose a novel framework, termed Knowledge Integration to HOI (KI2HOI), that effectively integrates the knowledge of visual-language model to improve zero-shot HOI detection. We develop an effective additive self-attention mechanism to generate more comprehensive visual representations. Our model outperforms the previous methods in various zero-shot and full-supervised settings.
  arXiv  Detail & Related papers  (2024-03-12T02:07:23Z)
• Detecting Any Human-Object Interaction Relationship: Universal HOI Detector with Spatial Prompt Learning on Foundation Models [55.20626448358655]
  This study explores the universal interaction recognition in an open-world setting through the use of Vision-Language (VL) foundation models and large language models (LLMs) Our design includes an HO Prompt-guided Decoder (HOPD), facilitates the association of high-level relation representations in the foundation model with various HO pairs within the image. For open-category interaction recognition, our method supports either of two input types: interaction phrase or interpretive sentence.
  arXiv  Detail & Related papers  (2023-11-07T08:27:32Z)
• Knowledge Graph Augmented Network Towards Multiview Representation Learning for Aspect-based Sentiment Analysis [96.53859361560505]
  We propose a knowledge graph augmented network (KGAN) to incorporate external knowledge with explicitly syntactic and contextual information. KGAN captures the sentiment feature representations from multiple perspectives, i.e., context-, syntax- and knowledge-based. Experiments on three popular ABSA benchmarks demonstrate the effectiveness and robustness of our KGAN.
  arXiv  Detail & Related papers  (2022-01-13T08:25:53Z)
• AU-Expression Knowledge Constrained Representation Learning for Facial Expression Recognition [79.8779790682205]
  We propose an AU-Expression Knowledge Constrained Representation Learning (AUE-CRL) framework to learn the AU representations without AU annotations and adaptively use representations to facilitate facial expression recognition. We conduct experiments on the challenging uncontrolled datasets to demonstrate the superiority of the proposed framework over current state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-12-29T03:42:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.