Guard Me If You Know Me: Protecting Specific Face-Identity from Deepfakes

• URL: http://arxiv.org/abs/2505.19582v1
• Date: Mon, 26 May 2025 06:55:23 GMT
• Title: Guard Me If You Know Me: Protecting Specific Face-Identity from Deepfakes
• Authors: Kaiqing Lin, Zhiyuan Yan, Ke-Yue Zhang, Li Hao, Yue Zhou, Yuzhen Lin, Weixiang Li, Taiping Yao, Shouhong Ding, Bin Li,
• Abstract summary: Securing personal identity against deepfake attacks is increasingly critical in the digital age.<n>Most existing deepfake detection methods focus on general-purpose scenarios.<n>We propose textbfVIPGuard, a unified multimodal framework for deepfake detection.
• Score: 37.12401429882299
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Securing personal identity against deepfake attacks is increasingly critical in the digital age, especially for celebrities and political figures whose faces are easily accessible and frequently targeted. Most existing deepfake detection methods focus on general-purpose scenarios and often ignore the valuable prior knowledge of known facial identities, e.g., "VIP individuals" whose authentic facial data are already available. In this paper, we propose \textbf{VIPGuard}, a unified multimodal framework designed to capture fine-grained and comprehensive facial representations of a given identity, compare them against potentially fake or similar-looking faces, and reason over these comparisons to make accurate and explainable predictions. Specifically, our framework consists of three main stages. First, fine-tune a multimodal large language model (MLLM) to learn detailed and structural facial attributes. Second, we perform identity-level discriminative learning to enable the model to distinguish subtle differences between highly similar faces, including real and fake variations. Finally, we introduce user-specific customization, where we model the unique characteristics of the target face identity and perform semantic reasoning via MLLM to enable personalized and explainable deepfake detection. Our framework shows clear advantages over previous detection works, where traditional detectors mainly rely on low-level visual cues and provide no human-understandable explanations, while other MLLM-based models often lack a detailed understanding of specific face identities. To facilitate the evaluation of our method, we built a comprehensive identity-aware benchmark called \textbf{VIPBench} for personalized deepfake detection, involving the latest 7 face-swapping and 7 entire face synthesis techniques for generation.

Related papers

• FaceShield: Explainable Face Anti-Spoofing with Multimodal Large Language Models [51.858371492494456]
  Face anti-spoofing (FAS) is crucial for protecting facial recognition systems from presentation attacks.<n>There is currently no universal and comprehensive MLLM and dataset specifically designed for FAS task.<n>We propose FaceShield, a MLLM for FAS, along with the corresponding pre-training and supervised fine-tuning datasets.<n>Our instruction datasets, protocols, and codes will be released soon.
  arXiv  Detail & Related papers  (2025-05-14T14:10:43Z)
• FaceInsight: A Multimodal Large Language Model for Face Perception [69.06084304620026]
  We propose FaceInsight, a versatile face perception large language model (MLLM) that provides fine-grained facial information.<n>Our approach introduces visual-textual alignment of facial knowledge to model both uncertain dependencies and deterministic relationships among facial information.<n> Comprehensive experiments and analyses across three face perception tasks demonstrate that FaceInsight consistently outperforms nine compared MLLMs.
  arXiv  Detail & Related papers  (2025-04-22T06:31:57Z)
• CLIP Unreasonable Potential in Single-Shot Face Recognition [0.0]
  Face recognition is a core task in computer vision designed to identify and authenticate individuals by analyzing facial patterns and features. Recent Contrastive Language Image Pretraining (CLIP) a model developed by OpenAI has shown promising advancements. CLIP links natural language processing with vision tasks allowing it to generalize across modalities.
  arXiv  Detail & Related papers  (2024-11-19T08:23:52Z)
• Face Anonymization Made Simple [44.24233169815565]
  Current face anonymization techniques often depend on identity loss calculated by face recognition models, which can be inaccurate and unreliable. In contrast, our approach uses diffusion models with only a reconstruction loss, eliminating the need for facial landmarks or masks. Our model achieves state-of-the-art performance in three key areas: identity anonymization, facial preservation, and image quality.
  arXiv  Detail & Related papers  (2024-11-01T17:45:21Z)
• LAFS: Landmark-based Facial Self-supervised Learning for Face Recognition [37.4550614524874]
  We focus on learning facial representations that can be adapted to train effective face recognition models. We explore the learning strategy of unlabeled facial images through self-supervised pretraining. Our method achieves significant improvement over the state-of-the-art on multiple face recognition benchmarks.
  arXiv  Detail & Related papers  (2024-03-13T01:07:55Z)
• Diff-Privacy: Diffusion-based Face Privacy Protection [58.1021066224765]
  In this paper, we propose a novel face privacy protection method based on diffusion models, dubbed Diff-Privacy. Specifically, we train our proposed multi-scale image inversion module (MSI) to obtain a set of SDM format conditional embeddings of the original image. Based on the conditional embeddings, we design corresponding embedding scheduling strategies and construct different energy functions during the denoising process to achieve anonymization and visual identity information hiding.
  arXiv  Detail & Related papers  (2023-09-11T09:26:07Z)
• OPOM: Customized Invisible Cloak towards Face Privacy Protection [58.07786010689529]
  We investigate the face privacy protection from a technology standpoint based on a new type of customized cloak. We propose a new method, named one person one mask (OPOM), to generate person-specific (class-wise) universal masks. The effectiveness of the proposed method is evaluated on both common and celebrity datasets.
  arXiv  Detail & Related papers  (2022-05-24T11:29:37Z)
• Master Face Attacks on Face Recognition Systems [45.090037010778765]
  Face authentication is now widely used, especially on mobile devices, rather than authentication using a personal identification number or an unlock pattern. Previous work has proven the existence of master faces that match multiple enrolled templates in face recognition systems. In this paper, we perform an extensive study on latent variable evolution (LVE), a method commonly used to generate master faces.
  arXiv  Detail & Related papers  (2021-09-08T02:11:35Z)
• A Systematical Solution for Face De-identification [6.244117712209321]
  In different tasks, people have various requirements for face de-identification (De-ID) We propose a systematical solution compatible for these De-ID operations. Our method can flexibly de-identify the face data in various ways and the processed images have high image quality.
  arXiv  Detail & Related papers  (2021-07-19T02:02:51Z)
• Towards Face Encryption by Generating Adversarial Identity Masks [53.82211571716117]
  We propose a targeted identity-protection iterative method (TIP-IM) to generate adversarial identity masks. TIP-IM provides 95%+ protection success rate against various state-of-the-art face recognition models.
  arXiv  Detail & Related papers  (2020-03-15T12:45:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.