CLIP-AUTT: Test-Time Personalization with Action Unit Prompting for Fine-Grained Video Emotion Recognition

• URL: http://arxiv.org/abs/2603.27999v2
• Date: Tue, 31 Mar 2026 15:59:48 GMT
• Title: CLIP-AUTT: Test-Time Personalization with Action Unit Prompting for Fine-Grained Video Emotion Recognition
• Abstract summary: Action Units (AUs) are structured textual prompts within CLIP to model fine-grained facial expressions.<n>We introduce CLIP-AU, a lightweight AU-guided temporal learning method that integrates interpretable AU semantics into CLIP.<n>We propose CLIP-AUTT, a video-based test-time personalization method that dynamically adapts AU prompts to videos from unseen subjects.
• Score: 57.8548595493709
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Personalization in emotion recognition (ER) is essential for an accurate interpretation of subtle and subject-specific expressive patterns. Recent advances in vision-language models (VLMs) such as CLIP demonstrate strong potential for leveraging joint image-text representations in ER. However, CLIP-based methods either depend on CLIP's contrastive pretraining or on LLMs to generate descriptive text prompts, which are noisy, computationally expensive, and fail to capture fine-grained expressions, leading to degraded performance. In this work, we leverage Action Units (AUs) as structured textual prompts within CLIP to model fine-grained facial expressions. AUs encode the subtle muscle activations underlying expressions, providing localized and interpretable semantic cues for more robust ER. We introduce CLIP-AU, a lightweight AU-guided temporal learning method that integrates interpretable AU semantics into CLIP. It learns generic, subject-agnostic representations by aligning AU prompts with facial dynamics, enabling fine-grained ER without CLIP fine-tuning or LLM-generated text supervision. Although CLIP-AU models fine-grained AU semantics, it does not adapt to subject-specific variability in subtle expressions. To address this limitation, we propose CLIP-AUTT, a video-based test-time personalization method that dynamically adapts AU prompts to videos from unseen subjects. By combining entropy-guided temporal window selection with prompt tuning, CLIP-AUTT enables subject-specific adaptation while preserving temporal consistency. Our extensive experiments on three challenging video-based subtle ER datasets, BioVid, StressID, and BAH, indicate that CLIP-AU and CLIP-AUTT outperform state-of-the-art CLIP-based FER and TTA methods, achieving robust and personalized subtle ER. Our code is publicly available at: https://github.com/osamazeeshan/CLIP-AUTT.

Related papers

• Fine-grained CLIP fine-tuning with self-annotated region alignment [57.28808560528696]
  Contrastive Language-Image Pre-training has been shown to have limitations in its fine-grained dense feature representation.<n>We propose SFF-CLIP (Self-annotated Fine-grained Fine-tuning for CLIP) to boost the fine-grained representation ability in the CLIP fine-tuning.
  arXiv  Detail & Related papers  (2026-07-15T10:06:39Z)
• ALADIN:Attribute-Language Distillation Network for Person Re-Identification [3.350310340720105]
  ALADIN is an attribute-language distillation network that distills knowledge from a frozen CLIP teacher to a lightweight ReID student.<n>A Scene-Aware Prompt Generator produces image-specific soft prompts to facilitate adaptive alignment.
  arXiv  Detail & Related papers  (2026-03-23T02:05:22Z)
• ProCLIP: Progressive Vision-Language Alignment via LLM-based Embedder [50.25233123718465]
  The CLIP text encoder is limited by a maximum input length of 77 tokens.<n>ProCLIP is a curriculum learning-based progressive vision-language alignment framework.
  arXiv  Detail & Related papers  (2025-10-21T16:48:49Z)
• Multimodal Prompt Alignment for Facial Expression Recognition [24.470095812039286]
  MPA-FER provides fine-grained semantic guidance to the learning process of prompted visual features.<n>Our framework outperforms state-of-the-art methods on three FER benchmark datasets.
  arXiv  Detail & Related papers  (2025-06-26T05:28:57Z)
• Open-Set Video-based Facial Expression Recognition with Human Expression-sensitive Prompting [28.673734895558322]
  We introduce a challenging Open-set Video-based Facial Expression Recognition task, aiming to identify both known and new, unseen facial expressions. Existing approaches use large-scale vision-language models like CLIP to identify unseen classes. We propose a novel Human Expression-Sensitive Prompting (HESP) mechanism to significantly enhance CLIP's ability to model video-based facial expression details effectively.
  arXiv  Detail & Related papers  (2024-04-26T01:21:08Z)
• FiGCLIP: Fine-Grained CLIP Adaptation via Densely Annotated Videos [19.08882495584709]
  We show that it is possible to enhance CLIP's fine-grained and syntactic abilities without compromising its semantic properties. We adapt CLIP efficiently on a high-quality, comprehensive, and relatively small dataset. We learn a powerful visual representation, dubbed Fine-Grained CLIP (FiGCLIP), that preserves semantic understanding while being detail-oriented.
  arXiv  Detail & Related papers  (2024-01-15T13:27:34Z)
• CLIP-DINOiser: Teaching CLIP a few DINO tricks for open-vocabulary semantic segmentation [31.264574799748903]
  We propose an open-vocabulary semantic segmentation method, which does not require any annotations. We show that the used self-supervised feature properties can directly be learnt from CLIP features. Our method CLIP-DINOiser needs only a single forward pass of CLIP and two light convolutional layers at inference.
  arXiv  Detail & Related papers  (2023-12-19T17:40:27Z)
• Bootstrap Fine-Grained Vision-Language Alignment for Unified Zero-Shot Anomaly Localization [63.61093388441298]
  Contrastive Language-Image Pre-training models have shown promising performance on zero-shot visual recognition tasks. In this work, we propose AnoCLIP for zero-shot anomaly localization.
  arXiv  Detail & Related papers  (2023-08-30T10:35:36Z)
• Prompting Visual-Language Models for Dynamic Facial Expression Recognition [12.51368195072919]
  This paper presents a novel visual-language model called DFER-CLIP.<n>It is based on the CLIP model and designed for in-the-wild Dynamic Facial Expression Recognition.<n>It achieves state-of-the-art results compared with the current supervised DFER methods on the DFEW, FERV39k, and MAFW benchmarks.
  arXiv  Detail & Related papers  (2023-08-25T13:52:05Z)
• CLIP4STR: A Simple Baseline for Scene Text Recognition with Pre-trained Vision-Language Model [55.321010757641524]
  We introduce CLIP4STR, a simple yet effective STR method built upon image and text encoders of CLIP.<n>We scale CLIP4STR in terms of the model size, pre-training data, and training data, achieving state-of-the-art performance on 13 STR benchmarks.
  arXiv  Detail & Related papers  (2023-05-23T12:51:20Z)
• CLIP-Adapter: Better Vision-Language Models with Feature Adapters [84.88106370842883]
  We show that there is an alternative path to achieve better vision-language models other than prompt tuning.<n>CLIP-Adapter adopts an additional bottleneck layer to learn new features and performs residual-style feature blending.<n> Experiments and extensive ablation studies on various visual classification tasks demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2021-10-09T11:39:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.