Semantic Contextualization of Face Forgery: A New Definition, Dataset, and Detection Method

• URL: http://arxiv.org/abs/2405.08487v3
• Date: Sat, 05 Apr 2025 09:19:17 GMT
• Title: Semantic Contextualization of Face Forgery: A New Definition, Dataset, and Detection Method
• Authors: Mian Zou, Baosheng Yu, Yibing Zhan, Siwei Lyu, Kede Ma,
• Abstract summary: We put face forgery in a semantic context and define that computational methods that alter semantic face attributes to exceed human discrimination thresholds are sources of face forgery.<n>We propose a semantics-oriented face forgery detection method that captures label relations and prioritizes the primary task (i.e., real or fake face detection)<n>We show that the proposed dataset successfully exposes the weaknesses of current detectors as the test set and consistently improves their generalizability as the training set.
• Score: 77.65459419417533
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In recent years, deep learning has greatly streamlined the process of manipulating photographic face images. Aware of the potential dangers, researchers have developed various tools to spot these counterfeits. Yet, none asks the fundamental question: What digital manipulations make a real photographic face image fake, while others do not? In this paper, we put face forgery in a semantic context and define that computational methods that alter semantic face attributes to exceed human discrimination thresholds are sources of face forgery. Following our definition, we construct a large face forgery image dataset, where each image is associated with a set of labels organized in a hierarchical graph. Our dataset enables two new testing protocols to probe the generalizability of face forgery detectors. Moreover, we propose a semantics-oriented face forgery detection method that captures label relations and prioritizes the primary task (i.e., real or fake face detection). We show that the proposed dataset successfully exposes the weaknesses of current detectors as the test set and consistently improves their generalizability as the training set. Additionally, we demonstrate the superiority of our semantics-oriented method over traditional binary and multi-class classification-based detectors.

Related papers

• Knowledge-Guided Prompt Learning for Deepfake Facial Image Detection [54.26588902144298]
  We propose a knowledge-guided prompt learning method for deepfake facial image detection. Specifically, we retrieve forgery-related prompts from large language models as expert knowledge to guide the optimization of learnable prompts. Our proposed approach notably outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-01-01T02:18:18Z)
• Semantics-Oriented Multitask Learning for DeepFake Detection: A Joint Embedding Approach [77.65459419417533]
  We propose an automatic dataset expansion technique to support semantics-oriented DeepFake detection tasks. We also resort to joint embedding of face images and their corresponding labels for prediction. Our method improves the generalizability of DeepFake detection and renders some degree of model interpretation by providing human-understandable explanations.
  arXiv  Detail & Related papers  (2024-08-29T07:11:50Z)
• UniForensics: Face Forgery Detection via General Facial Representation [60.5421627990707]
  High-level semantic features are less susceptible to perturbations and not limited to forgery-specific artifacts, thus having stronger generalization. We introduce UniForensics, a novel deepfake detection framework that leverages a transformer-based video network, with a meta-functional face classification for enriched facial representation.
  arXiv  Detail & Related papers  (2024-07-26T20:51:54Z)
• DiffusionFace: Towards a Comprehensive Dataset for Diffusion-Based Face Forgery Analysis [71.40724659748787]
  DiffusionFace is the first diffusion-based face forgery dataset. It covers various forgery categories, including unconditional and Text Guide facial image generation, Img2Img, Inpaint, and Diffusion-based facial exchange algorithms. It provides essential metadata and a real-world internet-sourced forgery facial image dataset for evaluation.
  arXiv  Detail & Related papers  (2024-03-27T11:32:44Z)
• DeepFidelity: Perceptual Forgery Fidelity Assessment for Deepfake Detection [67.3143177137102]
  Deepfake detection refers to detecting artificially generated or edited faces in images or videos. We propose a novel Deepfake detection framework named DeepFidelity to adaptively distinguish real and fake faces.
  arXiv  Detail & Related papers  (2023-12-07T07:19:45Z)
• Exploring Decision-based Black-box Attacks on Face Forgery Detection [53.181920529225906]
  Face forgery generation technologies generate vivid faces, which have raised public concerns about security and privacy. Although face forgery detection has successfully distinguished fake faces, recent studies have demonstrated that face forgery detectors are very vulnerable to adversarial examples.
  arXiv  Detail & Related papers  (2023-10-18T14:49:54Z)
• Building an Invisible Shield for Your Portrait against Deepfakes [34.65356811439098]
  We propose a novel framework - Integrity Encryptor, aiming to protect portraits in a proactive strategy. Our methodology involves covertly encoding messages that are closely associated with key facial attributes into authentic images. The modified facial attributes serve as a mean of detecting manipulated images through a comparison of the decoded messages.
  arXiv  Detail & Related papers  (2023-05-22T10:01:28Z)
• Disguise without Disruption: Utility-Preserving Face De-Identification [40.484745636190034]
  We introduce Disguise, a novel algorithm that seamlessly de-identifies facial images while ensuring the usability of the modified data. Our method involves extracting and substituting depicted identities with synthetic ones, generated using variational mechanisms to maximize obfuscation and non-invertibility. We extensively evaluate our method using multiple datasets, demonstrating a higher de-identification rate and superior consistency compared to prior approaches in various downstream tasks.
  arXiv  Detail & Related papers  (2023-03-23T13:50:46Z)
• A survey on facial image deblurring [3.6775758132528877]
  When the facial image is blurred, it has a great impact on high-level vision tasks such as face recognition. This paper surveys and summarizes recently published methods for facial image deblurring, most of which are based on deep learning. We show the performance of classical methods on datasets and metrics and give a brief discussion on the differences of model-based and learning-based methods.
  arXiv  Detail & Related papers  (2023-02-10T02:24:56Z)
• Detect Faces Efficiently: A Survey and Evaluations [13.105528567365281]
  Many applications including face recognition, facial expression recognition, face tracking and head-pose estimation assume that both the location and the size of faces are known in the image. Deep learning techniques brought remarkable breakthroughs to face detection along with the price of a considerable increase in computation. This paper introduces representative deep learning-based methods and presents a deep and thorough analysis in terms of accuracy and efficiency.
  arXiv  Detail & Related papers  (2021-12-03T08:39:40Z)
• DeepFake Detection Based on the Discrepancy Between the Face and its Context [94.47879216590813]
  We propose a method for detecting face swapping and other identity manipulations in single images. Our approach involves two networks: (i) a face identification network that considers the face region bounded by a tight semantic segmentation, and (ii) a context recognition network that considers the face context. We describe a method which uses the recognition signals from our two networks to detect such discrepancies. Our method achieves state of the art results on the FaceForensics++, Celeb-DF-v2, and DFDC benchmarks for face manipulation detection, and even generalizes to detect fakes produced by unseen methods.
  arXiv  Detail & Related papers  (2020-08-27T17:04:46Z)
• Fighting Deepfake by Exposing the Convolutional Traces on Images [0.0]
  Mobile apps like FACEAPP make use of the most advanced Generative Adversarial Networks (GAN) to produce extreme transformations on human face photos. This kind of media object took the name of Deepfake and raised a new challenge in the multimedia forensics field: the Deepfake detection challenge. In this paper, a new approach aimed to extract a Deepfake fingerprint from images is proposed.
  arXiv  Detail & Related papers  (2020-08-07T08:49:23Z)
• One-Shot GAN Generated Fake Face Detection [3.3707422585608953]
  We propose a universal One-Shot GAN generated fake face detection method. The proposed method is based on extracting out-of-context objects from faces via scene understanding models. Our experiments show that, we can discriminate fake faces from real ones in terms of out-of-context features.
  arXiv  Detail & Related papers  (2020-03-27T05:51:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.