Face-LLaVA: Facial Expression and Attribute Understanding through Instruction Tuning

• URL: http://arxiv.org/abs/2504.07198v1
• Date: Wed, 09 Apr 2025 18:26:07 GMT
• Title: Face-LLaVA: Facial Expression and Attribute Understanding through Instruction Tuning
• Authors: Ashutosh Chaubey, Xulang Guan, Mohammad Soleymani,
• Abstract summary: We propose Face-LLaVA, a large language model for face-centered, in-context learning, including facial expression and attribute recognition.<n>We first developed FaceInstruct-1M, a face-centered database for instruction tuning MLLMs for face processing.<n>We then developed a novel face-specific visual encoder powered by Face-Region Guided Cross-Attention.
• Score: 5.178801281905521
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: The human face plays a central role in social communication, necessitating the use of performant computer vision tools for human-centered applications. We propose Face-LLaVA, a multimodal large language model for face-centered, in-context learning, including facial expression and attribute recognition. Additionally, Face-LLaVA is able to generate natural language descriptions that can be used for reasoning. Leveraging existing visual databases, we first developed FaceInstruct-1M, a face-centered database for instruction tuning MLLMs for face processing. We then developed a novel face-specific visual encoder powered by Face-Region Guided Cross-Attention that integrates face geometry with local visual features. We evaluated the proposed method across nine different datasets and five different face processing tasks, including facial expression recognition, action unit detection, facial attribute detection, age estimation and deepfake detection. Face-LLaVA achieves superior results compared to existing open-source MLLMs and competitive performance compared to commercial solutions. Our model output also receives a higher reasoning rating by GPT under a zero-shot setting across all the tasks. Both our dataset and model wil be released at https://face-llava.github.io to support future advancements in social AI and foundational vision-language research.

Related papers

• FaceInsight: A Multimodal Large Language Model for Face Perception [69.06084304620026]
  We propose FaceInsight, a versatile face perception large language model (MLLM) that provides fine-grained facial information. Our approach introduces visual-textual alignment of facial knowledge to model both uncertain dependencies and deterministic relationships among facial information. Comprehensive experiments and analyses across three face perception tasks demonstrate that FaceInsight consistently outperforms nine compared MLLMs.
  arXiv  Detail & Related papers  (2025-04-22T06:31:57Z)
• OSDFace: One-Step Diffusion Model for Face Restoration [72.5045389847792]
  Diffusion models have demonstrated impressive performance in face restoration. We propose OSDFace, a novel one-step diffusion model for face restoration. Results demonstrate that OSDFace surpasses current state-of-the-art (SOTA) methods in both visual quality and quantitative metrics.
  arXiv  Detail & Related papers  (2024-11-26T07:07:48Z)
• Face-MLLM: A Large Face Perception Model [53.9441375205716]
  multimodal large language models (MLLMs) have achieved promising results on a wide range of vision-language tasks, but their ability to perceive and understand human faces is rarely explored. In this work, we comprehensively evaluate existing MLLMs on face perception tasks. Our model surpasses previous MLLMs on five famous face perception tasks.
  arXiv  Detail & Related papers  (2024-10-28T04:19:32Z)
• FaceXFormer: A Unified Transformer for Facial Analysis [59.94066615853198]
  FaceXFormer is an end-to-end unified transformer model capable of performing ten facial analysis tasks.<n>Tasks include face parsing, landmark detection, head pose estimation, attribute prediction, age, gender, and race estimation.<n>We train FaceXFormer on ten diverse face perception datasets and evaluate it against both specialized and multi-task models.
  arXiv  Detail & Related papers  (2024-03-19T17:58:04Z)
• A Generalist FaceX via Learning Unified Facial Representation [77.74407008931486]
  FaceX is a novel facial generalist model capable of handling diverse facial tasks simultaneously. Our versatile FaceX achieves competitive performance compared to elaborate task-specific models on popular facial editing tasks.
  arXiv  Detail & Related papers  (2023-12-31T17:41:48Z)
• Toward High Quality Facial Representation Learning [58.873356953627614]
  We propose a self-supervised pre-training framework, called Mask Contrastive Face (MCF) We use feature map of a pre-trained visual backbone as a supervision item and use a partially pre-trained decoder for mask image modeling. Our model achieves 0.932 NME_diag$ for AFLW-19 face alignment and 93.96 F1 score for LaPa face parsing.
  arXiv  Detail & Related papers  (2023-09-07T09:11:49Z)
• General Facial Representation Learning in a Visual-Linguistic Manner [45.92447707178299]
  We introduce a framework, called FaRL, for general Facial Representation Learning in a visual-linguistic manner. We show that FaRL achieves better transfer performance compared with previous pre-trained models. Our model surpasses the state-of-the-art methods on face analysis tasks including face parsing and face alignment.
  arXiv  Detail & Related papers  (2021-12-06T15:22:05Z)
• A Multi-resolution Approach to Expression Recognition in the Wild [9.118706387430883]
  We propose a multi-resolution approach to solve the Facial Expression Recognition task. We ground our intuition on the observation that often faces images are acquired at different resolutions. To our aim, we use a ResNet-like architecture, equipped with Squeeze-and-Excitation blocks, trained on the Affect-in-the-Wild 2 dataset.
  arXiv  Detail & Related papers  (2021-03-09T21:21:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.