Task-Specific Feature Fusion Method for Multi-Task Affective Behavior Analysis

• URL: http://arxiv.org/abs/2607.13986v1
• Date: Wed, 15 Jul 2026 16:14:59 GMT
• Title: Task-Specific Feature Fusion Method for Multi-Task Affective Behavior Analysis
• Abstract summary: We study task-adaptive feature fusion for ABAW11 multi-task affective behavior analysis.<n>We first adapt two pretrained visual backbones, DINOv2 ViT-L and DINOv3 ConvNeXt-base, on an external expression-oriented facial image set.<n>On the ABAW11 validation set, the selected system achieves an EXPR macro-F1 of 0.4222, an AU macro-F1 of 0.5402, and a mean VA CCC of 0.6717.
• Score: 5.014305646574725
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The 11th Affective Behavior Analysis in-the-wild (ABAW11) Multi-Task Learning Challenge requires a unified system to predict valence-arousal, categorical expressions, and facial action units from the official s-Aff-Wild2 images. Although these tasks are naturally related through facial behavior, our validation experiments show that they benefit from different visual features, temporal processing strategies, fusion mechanisms, and calibration procedures. In this paper, we study task-adaptive feature fusion for ABAW11 multi-task affective behavior analysis. We first adapt two pretrained visual backbones, DINOv2 ViT-L and DINOv3 ConvNeXt-base, on an external expression-oriented facial image set and then freeze them to extract complementary frame-level features from the official ABAW11 data. On top of these frozen features, we systematically compare frame-level prediction heads, temporal convolutional heads, post-hoc temporal smoothing, LightGBM models, feature concatenation, gated fusion, residual fusion, late logit fusion, threshold calibration, and shared MTL structures. The final system selects task-specific fusion and prediction strategies rather than forcing all tasks to share a single architecture. On the ABAW11 validation set, the selected system achieves an EXPR macro-F1 of 0.4222, an AU macro-F1 of 0.5402, and a mean VA CCC of 0.6717, resulting in an overall validation score of 1.6341. The results suggest that task-adaptive fusion of frozen visual features is a simple and effective strategy for ABAW-style multi-task affective behavior analysis.

Related papers

• AffectFuse: Cross-Task Feature Fusion with Temporal Modeling for Multi-Task Affective Behavior Analysis [0.2981315866322933]
  We present our system for the Multi-Task Learning track of the 11th Affective Behavior Analysis in-the-wild (ABAW) competition on s-Aff-Wild2.<n>The method focuses on post-encoder adaptation: frozen AffectNet-supervised backbones provide multi-resolution features.<n>For action-unit recognition, we adapt MAE-Face with Low-Rank Adaptation (LoRA) and use DISFA through per-unit expert routing.
  arXiv  Detail & Related papers  (2026-07-17T22:44:43Z)
• From Insight to Action: A Novel Framework for Interpretability-Guided Data Selection in Large Language Models [73.72877445629383]
  Interpretability-Guided Data Selection (IGDS) is a framework that first identifies causal task features through frequency recall and interventional filtering.<n>We validate IGDS on mathematical reasoning, summarization, and translation tasks within Gemma-2, LLaMA-3.1, and Qwen3 models.
  arXiv  Detail & Related papers  (2026-04-28T03:16:24Z)
• Team RAS in 10th ABAW Competition: Multimodal Valence and Arousal Estimation Approach [53.64487844936037]
  Our method combines three complementary modalities: face, behavior, and audio.<n>Experiments demonstrate that the proposed multimodal fusion strategy achieves a Concordance Correlation Coefficient (CCC) of 0.658 on the Aff-Wild2 development set.
  arXiv  Detail & Related papers  (2026-03-13T15:06:14Z)
• CtrlFuse: Mask-Prompt Guided Controllable Infrared and Visible Image Fusion [51.060328159429154]
  Infrared and visible image fusion generates all-weather perception-capable images by combining complementary modalities.<n>We propose CtrlFuse, a controllable image fusion framework that enables interactive dynamic fusion guided by mask prompts.<n> Experiments demonstrate state-of-the-art results in both fusion controllability and segmentation accuracy, with the adapted task branch even outperforming the original segmentation model.
  arXiv  Detail & Related papers  (2026-01-12T13:36:48Z)
• Quantizing Space and Time: Fusing Time Series and Images for Earth Observation [4.012968772806928]
  We propose a task-agnostic framework for multimodal fusion of time series and single timestamp images.<n>Our approach explores deterministic and learned strategies for time series quantization.<n>Our model generates consistent global temperature profiles from satellite imagery.
  arXiv  Detail & Related papers  (2025-10-27T08:38:52Z)
• Affective Behaviour Analysis via Progressive Learning [23.455163723584427]
  The 7th Affective Behavior Analysis in-the-wild (ABAW) competition holds the Multi-Task Learning Challenge based on the s-Aff-Wild2 database.<n>The participants are required to develop a framework that achieves Valence-Arousal Estimation, Expression Recognition, and AU detection simultaneously.<n>We propose a progressive multi-task learning framework that fully leverages the distinct focuses of each task on facial emotion features.
  arXiv  Detail & Related papers  (2024-07-24T02:24:21Z)
• HSEmotion Team at the 7th ABAW Challenge: Multi-Task Learning and Compound Facial Expression Recognition [16.860963320038902]
  We describe the results of the HSEmotion team in two tasks of the seventh Affective Behavior Analysis in-the-wild (ABAW) competition. We propose an efficient pipeline based on frame-level facial feature extractors pre-trained in multi-task settings. We ensure the privacy-awareness of our techniques by using the lightweight architectures of neural networks.
  arXiv  Detail & Related papers  (2024-07-18T05:47:49Z)
• SwinFace: A Multi-task Transformer for Face Recognition, Expression Recognition, Age Estimation and Attribute Estimation [60.94239810407917]
  This paper presents a multi-purpose algorithm for simultaneous face recognition, facial expression recognition, age estimation, and face attribute estimation based on a single Swin Transformer. To address the conflicts among multiple tasks, a Multi-Level Channel Attention (MLCA) module is integrated into each task-specific analysis. Experiments show that the proposed model has a better understanding of the face and achieves excellent performance for all tasks.
  arXiv  Detail & Related papers  (2023-08-22T15:38:39Z)
• Self-supervision and Learnable STRFs for Age, Emotion, and Country Prediction [26.860736835176617]
  This work presents a multitask approach to the simultaneous estimation of age, country of origin, and emotion given vocal burst audio. We evaluate the complementarity between the tasks posed by examining independent task-specific and joint models, and explore the relative strengths of different feature sets. We find that robust data preprocessing in conjunction with score fusion over spectro-temporal receptive field and HuBERT models achieved our best ExVo-MultiTask test score of 0.412.
  arXiv  Detail & Related papers  (2022-06-25T06:09:10Z)
• Pluralistic Image Completion with Probabilistic Mixture-of-Experts [58.81469985455467]
  We introduce a unified probabilistic graph model that represents the complex interactions in image completion. The entire procedure of image completion is then mathematically divided into several sub-procedures, which helps efficient enforcement of constraints. The inherent parameters of GMM are task-related, which are optimized adaptively during training, while the number of its primitives can control the diversity of results conveniently.
  arXiv  Detail & Related papers  (2022-05-18T17:24:21Z)
• Hybrid Relation Guided Set Matching for Few-shot Action Recognition [51.3308583226322]
  We propose a novel Hybrid Relation guided Set Matching (HyRSM) approach that incorporates two key components. The purpose of the hybrid relation module is to learn task-specific embeddings by fully exploiting associated relations within and cross videos in an episode. We evaluate HyRSM on six challenging benchmarks, and the experimental results show its superiority over the state-of-the-art methods by a convincing margin.
  arXiv  Detail & Related papers  (2022-04-28T11:43:41Z)
• Pruning Self-attentions into Convolutional Layers in Single Path [89.55361659622305]
  Vision Transformers (ViTs) have achieved impressive performance over various computer vision tasks. We propose Single-Path Vision Transformer pruning (SPViT) to efficiently and automatically compress the pre-trained ViTs. Our SPViT can trim 52.0% FLOPs for DeiT-B and get an impressive 0.6% top-1 accuracy gain simultaneously.
  arXiv  Detail & Related papers  (2021-11-23T11:35:54Z)
• A Multi-term and Multi-task Analyzing Framework for Affective Analysis in-the-wild [0.2216657815393579]
  We introduce the affective recognition method that was submitted to the Affective Behavior Analysis in-the-wild (ABAW) 2020 Contest. Since affective behaviors have many observable features that have their own time frames, we introduced multiple optimized time windows. We generated affective recognition models for each time window and ensembled these models together.
  arXiv  Detail & Related papers  (2020-09-29T09:24:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.