FaceInsight: A Multimodal Large Language Model for Face Perception

• URL: http://arxiv.org/abs/2504.15624v2
• Date: Fri, 25 Apr 2025 18:58:49 GMT
• Title: FaceInsight: A Multimodal Large Language Model for Face Perception
• Authors: Jingzhi Li, Changjiang Luo, Ruoyu Chen, Hua Zhang, Wenqi Ren, Jianhou Gan, Xiaochun Cao,
• Abstract summary: We propose FaceInsight, a versatile face perception large language model (MLLM) that provides fine-grained facial information.<n>Our approach introduces visual-textual alignment of facial knowledge to model both uncertain dependencies and deterministic relationships among facial information.<n> Comprehensive experiments and analyses across three face perception tasks demonstrate that FaceInsight consistently outperforms nine compared MLLMs.
• Score: 69.06084304620026
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Recent advances in multimodal large language models (MLLMs) have demonstrated strong capabilities in understanding general visual content. However, these general-domain MLLMs perform poorly in face perception tasks, often producing inaccurate or misleading responses to face-specific queries. To address this gap, we propose FaceInsight, the versatile face perception MLLM that provides fine-grained facial information. Our approach introduces visual-textual alignment of facial knowledge to model both uncertain dependencies and deterministic relationships among facial information, mitigating the limitations of language-driven reasoning. Additionally, we incorporate face segmentation maps as an auxiliary perceptual modality, enriching the visual input with localized structural cues to enhance semantic understanding. Comprehensive experiments and analyses across three face perception tasks demonstrate that FaceInsight consistently outperforms nine compared MLLMs under both training-free and fine-tuned settings.

Related papers

• Face-LLaVA: Facial Expression and Attribute Understanding through Instruction Tuning [5.178801281905521]
  We propose Face-LLaVA, a large language model for face-centered, in-context learning, including facial expression and attribute recognition.<n>We first developed FaceInstruct-1M, a face-centered database for instruction tuning MLLMs for face processing.<n>We then developed a novel face-specific visual encoder powered by Face-Region Guided Cross-Attention.
  arXiv  Detail & Related papers  (2025-04-09T18:26:07Z)
• Interpretable Face Anti-Spoofing: Enhancing Generalization with Multimodal Large Language Models [58.936893810674896]
  Face Anti-Spoofing (FAS) is essential for ensuring the security and reliability of facial recognition systems.<n>We introduce a multimodal large language model framework for FAS, termed Interpretable Face Anti-Spoofing (I-FAS)<n>We propose a Spoof-aware Captioning and Filtering (SCF) strategy to generate high-quality captions for FAS images.
  arXiv  Detail & Related papers  (2025-01-03T09:25:04Z)
• Face-MLLM: A Large Face Perception Model [53.9441375205716]
  multimodal large language models (MLLMs) have achieved promising results on a wide range of vision-language tasks, but their ability to perceive and understand human faces is rarely explored. In this work, we comprehensively evaluate existing MLLMs on face perception tasks. Our model surpasses previous MLLMs on five famous face perception tasks.
  arXiv  Detail & Related papers  (2024-10-28T04:19:32Z)
• Rethinking Visual Prompting for Multimodal Large Language Models with External Knowledge [76.45868419402265]
  multimodal large language models (MLLMs) have made significant strides by training on vast high-quality image-text datasets. However, the inherent difficulty in explicitly conveying fine-grained or spatially dense information in text, such as masks, poses a challenge for MLLMs. This paper proposes a new visual prompt approach to integrate fine-grained external knowledge, gleaned from specialized vision models, into MLLMs.
  arXiv  Detail & Related papers  (2024-07-05T17:43:30Z)
• Text-Guided Face Recognition using Multi-Granularity Cross-Modal Contrastive Learning [0.0]
  We introduce text-guided face recognition (TGFR) to analyze the impact of integrating facial attributes in the form of natural language descriptions. TGFR demonstrates remarkable improvements, particularly on low-quality images, over existing face recognition models.
  arXiv  Detail & Related papers  (2023-12-14T22:04:22Z)
• GaFET: Learning Geometry-aware Facial Expression Translation from In-The-Wild Images [55.431697263581626]
  We introduce a novel Geometry-aware Facial Expression Translation framework, which is based on parametric 3D facial representations and can stably decoupled expression. We achieve higher-quality and more accurate facial expression transfer results compared to state-of-the-art methods, and demonstrate applicability of various poses and complex textures.
  arXiv  Detail & Related papers  (2023-08-07T09:03:35Z)
• CLIPER: A Unified Vision-Language Framework for In-the-Wild Facial Expression Recognition [1.8604727699812171]
  We propose a unified framework for both static and dynamic facial Expression Recognition based on CLIP. We introduce multiple expression text descriptors (METD) to learn fine-grained expression representations that make CLIPER more interpretable.
  arXiv  Detail & Related papers  (2023-03-01T02:59:55Z)
• General Facial Representation Learning in a Visual-Linguistic Manner [45.92447707178299]
  We introduce a framework, called FaRL, for general Facial Representation Learning in a visual-linguistic manner. We show that FaRL achieves better transfer performance compared with previous pre-trained models. Our model surpasses the state-of-the-art methods on face analysis tasks including face parsing and face alignment.
  arXiv  Detail & Related papers  (2021-12-06T15:22:05Z)
• I Only Have Eyes for You: The Impact of Masks On Convolutional-Based Facial Expression Recognition [78.07239208222599]
  We evaluate how the recently proposed FaceChannel adapts towards recognizing facial expressions from persons with masks. We also perform specific feature-level visualization to demonstrate how the inherent capabilities of the FaceChannel to learn and combine facial features change when in a constrained social interaction scenario.
  arXiv  Detail & Related papers  (2021-04-16T20:03:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.