HAMLET-FFD: Hierarchical Adaptive Multi-modal Learning Embeddings Transformation for Face Forgery Detection

• URL: http://arxiv.org/abs/2507.20913v1
• Date: Mon, 28 Jul 2025 15:09:52 GMT
• Title: HAMLET-FFD: Hierarchical Adaptive Multi-modal Learning Embeddings Transformation for Face Forgery Detection
• Authors: Jialei Cui, Jianwei Du, Yanzhe Li, Lei Gao, Hui Jiang, Chenfu Bao,
• Abstract summary: HAMLET-FFD is a cross-domain generalization framework for face forgery detection.<n>It integrates visual evidence with conceptual cues, emulating expert forensic analysis.<n>By design, HAMLET-FFD freezes all pretrained parameters, serving as an external plugin.
• Score: 6.060036926093259
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The rapid evolution of face manipulation techniques poses a critical challenge for face forgery detection: cross-domain generalization. Conventional methods, which rely on simple classification objectives, often fail to learn domain-invariant representations. We propose HAMLET-FFD, a cognitively inspired Hierarchical Adaptive Multi-modal Learning framework that tackles this challenge via bidirectional cross-modal reasoning. Building on contrastive vision-language models such as CLIP, HAMLET-FFD introduces a knowledge refinement loop that iteratively assesses authenticity by integrating visual evidence with conceptual cues, emulating expert forensic analysis. A key innovation is a bidirectional fusion mechanism in which textual authenticity embeddings guide the aggregation of hierarchical visual features, while modulated visual features refine text embeddings to generate image-adaptive prompts. This closed-loop process progressively aligns visual observations with semantic priors to enhance authenticity assessment. By design, HAMLET-FFD freezes all pretrained parameters, serving as an external plugin that preserves CLIP's original capabilities. Extensive experiments demonstrate its superior generalization to unseen manipulations across multiple benchmarks, and visual analyses reveal a division of labor among embeddings, with distinct representations specializing in fine-grained artifact recognition.

Related papers

• Multimodal Prompt Alignment for Facial Expression Recognition [24.470095812039286]
  MPA-FER provides fine-grained semantic guidance to the learning process of prompted visual features.<n>Our framework outperforms state-of-the-art methods on three FER benchmark datasets.
  arXiv  Detail & Related papers  (2025-06-26T05:28:57Z)
• BMRL: Bi-Modal Guided Multi-Perspective Representation Learning for Zero-Shot Deepfake Attribution [19.78648266444095]
  We propose a novel framework for zero-shot deepfake attribution (ZS-DFA)<n>Specifically, we design a multi-perspective visual encoder (MPVE) to explore general deepfake attribution visual characteristics across three views.<n>A language encoder is proposed to capture fine-grained language embeddings, facilitating language-guided general visual representation learning.
  arXiv  Detail & Related papers  (2025-04-19T01:11:46Z)
• "Principal Components" Enable A New Language of Images [79.45806370905775]
  We introduce a novel visual tokenization framework that embeds a provable PCA-like structure into the latent token space.<n>Our approach achieves state-of-the-art reconstruction performance and enables better interpretability to align with the human vision system.
  arXiv  Detail & Related papers  (2025-03-11T17:59:41Z)
• Exploring Interpretability for Visual Prompt Tuning with Hierarchical Concepts [39.92376420375139]
  We propose the first framework, named Interpretable Visual Prompt Tuning, to explore interpretability for visual prompts.<n>Visual prompts are linked to human-understandable semantic concepts, represented as a set of category-agnostic prototypes.<n>IVPT aggregates features from these regions to generate interpretable prompts, which are structured hierarchically to explain visual prompts at different granularities.
  arXiv  Detail & Related papers  (2025-03-08T06:12:50Z)
• MFCLIP: Multi-modal Fine-grained CLIP for Generalizable Diffusion Face Forgery Detection [64.29452783056253]
  The rapid development of photo-realistic face generation methods has raised significant concerns in society and academia.<n>Although existing approaches mainly capture face forgery patterns using image modality, other modalities like fine-grained noises and texts are not fully explored.<n>We propose a novel multi-modal fine-grained CLIP (MFCLIP) model, which mines comprehensive and fine-grained forgery traces across image-noise modalities.
  arXiv  Detail & Related papers  (2024-09-15T13:08:59Z)
• UniForensics: Face Forgery Detection via General Facial Representation [60.5421627990707]
  High-level semantic features are less susceptible to perturbations and not limited to forgery-specific artifacts, thus having stronger generalization. We introduce UniForensics, a novel deepfake detection framework that leverages a transformer-based video network, with a meta-functional face classification for enriched facial representation.
  arXiv  Detail & Related papers  (2024-07-26T20:51:54Z)
• CFPL-FAS: Class Free Prompt Learning for Generalizable Face Anti-spoofing [66.6712018832575]
  Domain generalization (DG) based Face Anti-Spoofing (FAS) aims to improve the model's performance on unseen domains. We make use of large-scale VLMs like CLIP and leverage the textual feature to dynamically adjust the classifier's weights for exploring generalizable visual features.
  arXiv  Detail & Related papers  (2024-03-21T11:58:50Z)
• Emotic Masked Autoencoder with Attention Fusion for Facial Expression Recognition [1.4374467687356276]
  This paper presents an innovative approach integrating the MAE-Face self-supervised learning (SSL) method and multi-view Fusion Attention mechanism for expression classification. We suggest easy-to-implement and no-training frameworks aimed at highlighting key facial features to determine if such features can serve as guides for the model. The efficacy of this method is validated by improvements in model performance on the Aff-wild2 dataset.
  arXiv  Detail & Related papers  (2024-03-19T16:21:47Z)
• Exploiting Modality-Specific Features For Multi-Modal Manipulation Detection And Grounding [54.49214267905562]
  We construct a transformer-based framework for multi-modal manipulation detection and grounding tasks. Our framework simultaneously explores modality-specific features while preserving the capability for multi-modal alignment. We propose an implicit manipulation query (IMQ) that adaptively aggregates global contextual cues within each modality.
  arXiv  Detail & Related papers  (2023-09-22T06:55:41Z)
• Towards General Visual-Linguistic Face Forgery Detection [95.73987327101143]
  Deepfakes are realistic face manipulations that can pose serious threats to security, privacy, and trust. Existing methods mostly treat this task as binary classification, which uses digital labels or mask signals to train the detection model. We propose a novel paradigm named Visual-Linguistic Face Forgery Detection(VLFFD), which uses fine-grained sentence-level prompts as the annotation.
  arXiv  Detail & Related papers  (2023-07-31T10:22:33Z)
• FER-former: Multi-modal Transformer for Facial Expression Recognition [14.219492977523682]
  A novel multifarious supervision-steering Transformer for Facial Expression Recognition is proposed in this paper. Our approach features multi-granularity embedding integration, hybrid self-attention scheme, and heterogeneous domain-steering supervision. Experiments on popular benchmarks demonstrate the superiority of the proposed FER-former over the existing state-of-the-arts.
  arXiv  Detail & Related papers  (2023-03-23T02:29:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.