Cross-Task Multi-Branch Vision Transformer for Facial Expression and Mask Wearing Classification

• URL: http://arxiv.org/abs/2404.14606v2
• Date: Tue, 30 Apr 2024 06:34:16 GMT
• Title: Cross-Task Multi-Branch Vision Transformer for Facial Expression and Mask Wearing Classification
• Authors: Armando Zhu, Keqin Li, Tong Wu, Peng Zhao, Bo Hong,
• Abstract summary: We propose a unified multi-branch vision transformer for facial expression recognition and mask wearing classification tasks. Our approach extracts shared features for both tasks using a dual-branch architecture. Our proposed framework reduces the overall complexity compared with using separate networks for both tasks.
• Score: 13.995453649985732
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: With wearing masks becoming a new cultural norm, facial expression recognition (FER) while taking masks into account has become a significant challenge. In this paper, we propose a unified multi-branch vision transformer for facial expression recognition and mask wearing classification tasks. Our approach extracts shared features for both tasks using a dual-branch architecture that obtains multi-scale feature representations. Furthermore, we propose a cross-task fusion phase that processes tokens for each task with separate branches, while exchanging information using a cross attention module. Our proposed framework reduces the overall complexity compared with using separate networks for both tasks by the simple yet effective cross-task fusion phase. Extensive experiments demonstrate that our proposed model performs better than or on par with different state-of-the-art methods on both facial expression recognition and facial mask wearing classification task.

Related papers

• Look Through Masks: Towards Masked Face Recognition with De-Occlusion Distillation [39.159835055226274]
  We propose to migrate the mechanism of amodal completion for the task of masked face recognition with an end-to-end de-occlusion distillation framework. The textitde-occlusion module applies a generative adversarial network to perform face completion, which recovers the content under the mask and eliminates appearance ambiguity. The textitdistillation module takes a pre-trained general face recognition model as the teacher and transfers its knowledge to train a student for completed faces.
  arXiv  Detail & Related papers  (2024-09-19T01:00:36Z)
• Comprehensive Generative Replay for Task-Incremental Segmentation with Concurrent Appearance and Semantic Forgetting [49.87694319431288]
  Generalist segmentation models are increasingly favored for diverse tasks involving various objects from different image sources. We propose a Comprehensive Generative (CGR) framework that restores appearance and semantic knowledge by synthesizing image-mask pairs. Experiments on incremental tasks (cardiac, fundus and prostate segmentation) show its clear advantage for alleviating concurrent appearance and semantic forgetting.
  arXiv  Detail & Related papers  (2024-06-28T10:05:58Z)
• FaceXFormer: A Unified Transformer for Facial Analysis [59.94066615853198]
  FaceXformer is an end-to-end unified transformer model for a range of facial analysis tasks. Our model effectively handles images "in-the-wild," demonstrating its robustness and generalizability across eight different tasks.
  arXiv  Detail & Related papers  (2024-03-19T17:58:04Z)
• A Simple Latent Diffusion Approach for Panoptic Segmentation and Mask Inpainting [2.7563282688229664]
  This work builds upon Stable Diffusion and proposes a latent diffusion approach for panoptic segmentation. Our training consists of two steps: (1) training a shallow autoencoder to project the segmentation masks to latent space; (2) training a diffusion model to allow image-conditioned sampling in latent space.
  arXiv  Detail & Related papers  (2024-01-18T18:59:19Z)
• Seeing through the Mask: Multi-task Generative Mask Decoupling Face Recognition [47.248075664420874]
  Current general face recognition system suffers from serious performance degradation when encountering occluded scenes. This paper proposes a Multi-task gEnerative mask dEcoupling face Recognition (MEER) network to jointly handle these two tasks. We first present a novel mask decoupling module to disentangle mask and identity information, which makes the network obtain purer identity features from visible facial components.
  arXiv  Detail & Related papers  (2023-11-20T03:23:03Z)
• SwinFace: A Multi-task Transformer for Face Recognition, Expression Recognition, Age Estimation and Attribute Estimation [60.94239810407917]
  This paper presents a multi-purpose algorithm for simultaneous face recognition, facial expression recognition, age estimation, and face attribute estimation based on a single Swin Transformer. To address the conflicts among multiple tasks, a Multi-Level Channel Attention (MLCA) module is integrated into each task-specific analysis. Experiments show that the proposed model has a better understanding of the face and achieves excellent performance for all tasks.
  arXiv  Detail & Related papers  (2023-08-22T15:38:39Z)
• Self-Supervised Visual Representations Learning by Contrastive Mask Prediction [129.25459808288025]
  We propose a novel contrastive mask prediction (CMP) task for visual representation learning. MaskCo contrasts region-level features instead of view-level features, which makes it possible to identify the positive sample without any assumptions. We evaluate MaskCo on training datasets beyond ImageNet and compare its performance with MoCo V2.
  arXiv  Detail & Related papers  (2021-08-18T02:50:33Z)
• Self-supervised Contrastive Learning of Multi-view Facial Expressions [9.949781365631557]
  Facial expression recognition (FER) has emerged as an important component of human-computer interaction systems. We propose Contrastive Learning of Multi-view facial Expressions (CL-MEx) to exploit facial images captured simultaneously from different angles towards FER.
  arXiv  Detail & Related papers  (2021-08-15T11:23:34Z)
• FT-TDR: Frequency-guided Transformer and Top-Down Refinement Network for Blind Face Inpainting [77.78305705925376]
  Blind face inpainting refers to the task of reconstructing visual contents without explicitly indicating the corrupted regions in a face image. We propose a novel two-stage blind face inpainting method named Frequency-guided Transformer and Top-Down Refinement Network (FT-TDR) to tackle these challenges.
  arXiv  Detail & Related papers  (2021-08-10T03:12:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.