Harnessing CLIP and DINO: An Uncertainty-Aware Cascaded Fusion Network for Generalizable Deepfake Image Detection

• URL: http://arxiv.org/abs/2609.07670v1
• Date: Mon, 07 Sep 2026 15:59:19 GMT
• Title: Harnessing CLIP and DINO: An Uncertainty-Aware Cascaded Fusion Network for Generalizable Deepfake Image Detection
• Abstract summary: UCF-Net is an uncertainty-aware cascaded fusion network that harnesses CLIP's language-aligned semantic priors and DINO's self-supervised visual-structure priors.<n>We consolidate public deepfake datasets into a unified benchmark of approximately 4M images and construct a separate cross-generator evaluation set with over 8K face images.<n>On the unified benchmark, UCF-Net achieves the best mean AUC among the evaluated methods in both in-domain and cross-domain evaluations.
• Score: 37.90071216560782
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The growing realism and accessibility of manipulated and generated faces threaten the trustworthiness of digital media. To detect such forgeries, deepfake detectors based on vision foundation models have shown promising performance, but they typically rely on a single pretrained representation and are prone to overfitting to particular training distributions. To improve generalization to unseen forgeries, we propose UCF-Net, an uncertainty-aware cascaded fusion network that harnesses CLIP's language-aligned semantic priors and DINO's self-supervised visual-structure priors. UCF-Net extracts hierarchical features across Transformer depths, uses layer-wise expert aggregation to adaptively combine each encoder's multi-level cues, and performs weighted fusion of the resulting representations based on entropy-derived uncertainty. We further consolidate public deepfake datasets into a unified benchmark of approximately 4M images and construct a separate cross-generator evaluation set with over 8K face images from eight recent generators. On the unified benchmark, UCF-Net achieves the best mean AUC among the evaluated methods in both in-domain and cross-domain evaluations. On the cross-generator set, it adapts effectively with limited target-domain data, although zero-shot transfer remains challenging.

Related papers

• Where Does Generative Difficulty Reside? An Empirical Study of Target Representations [62.54876287800644]
  The target representation defines the distribution an image generator must learn, yet it is often treated as an interchangeable interface.<n>We study raw pixels, SD-VAE latents and DINOv2 as well as MAE representation-autoencoder features within a unified masked autoregressive rectified-flow model.<n>Our results show that compression, reconstruction fidelity, token dimensionality, and visible semantic clustering do not individually predict generative behavior.
  arXiv  Detail & Related papers  (2026-08-01T12:32:31Z)
• Deepfake Forensics Adapter: A Dual-Stream Network for Generalizable Deepfake Detection [22.889849855283355]
  Deepfake Forensics Adapter (DFA) is a novel dual-stream framework that synergizes vision-language foundation models with targeted forensics analysis.<n>Our approach integrates a pre-trained CLIP model with three core components to achieve specialized deepfake detection.<n>Our framework not only demonstrates state-of-the-art performance, but also points out a feasible and effective direction for developing a robust deepfake detection system.
  arXiv  Detail & Related papers  (2026-03-02T04:58:00Z)
• Detecting AI-Generated Forgeries via Iterative Manifold Deviation Amplification [11.465428962772245]
  We propose IFA-Net, which shifts from learning "what is fake" to modeling "what is real"<n>Experiments on four diffusion-based inpainting benchmarks show that IFA-Net achieves an average improvement of 6.5% in IoU and 8.1% in F1-score over the second-best method.
  arXiv  Detail & Related papers  (2026-02-21T14:14:17Z)
• UAGLNet: Uncertainty-Aggregated Global-Local Fusion Network with Cooperative CNN-Transformer for Building Extraction [83.48950950780554]
  Building extraction from remote sensing images is a challenging task due to the complex structure variations of buildings.<n>Existing methods employ convolutional or self-attention blocks to capture the multi-scale features in the segmentation models.<n>We present an Uncertainty-Aggregated Global-Local Fusion Network (UAGLNet) to exploit high-quality global-local visual semantics.
  arXiv  Detail & Related papers  (2025-12-15T02:59:16Z)
• Multi-modal Deepfake Detection and Localization with FPN-Transformer [21.022230340898556]
  We introduce a multi-modal deepfake detection and localization framework based on a Feature Pyramid-Transformer (FPN-Transformer)<n>A multi-scale feature pyramid is constructed through R-TLM blocks with localized attention mechanisms, enabling joint analysis of cross-context temporal dependencies.<n>We evaluate our approach on the test set of the IJCAI'25 DDL-AV benchmark, showing a good performance with a final score of 0.7535.
  arXiv  Detail & Related papers  (2025-11-11T09:33:39Z)
• Knowledge-Informed Neural Network for Complex-Valued SAR Image Recognition [51.03674130115878]
  We introduce the Knowledge-Informed Neural Network (KINN), a lightweight framework built upon a novel "compression-aggregation-compression" architecture.<n>KINN establishes a state-of-the-art in parameter-efficient recognition, offering exceptional generalization in data-scarce and out-of-distribution scenarios.
  arXiv  Detail & Related papers  (2025-10-23T07:12:26Z)
• CAE-Net: Generalized Deepfake Image Detection using Convolution and Attention Mechanisms with Spatial and Frequency Domain Features [0.6700983301090583]
  We propose a disjoint set-based multistage training method to address the class imbalance and devised an ensemble-based architecture emphCAE-Net.<n>Our architecture consists of a convolution- and attention-based ensemble network, and employs three different neural network architectures.<n>Individually, the EfficientNet B0 architecture has achieved 90.79% accuracy, whereas the ConvNeXt and the DeiT architecture have achieved 89.49% and 89.32% accuracy, respectively.
  arXiv  Detail & Related papers  (2025-02-15T06:02:11Z)
• Contrasting Deepfakes Diffusion via Contrastive Learning and Global-Local Similarities [88.398085358514]
  Contrastive Deepfake Embeddings (CoDE) is a novel embedding space specifically designed for deepfake detection. CoDE is trained via contrastive learning by additionally enforcing global-local similarities.
  arXiv  Detail & Related papers  (2024-07-29T18:00:10Z)
• Cross-domain Robust Deepfake Bias Expansion Network for Face Forgery Detection [4.269822517578155]
  We introduce a Cross-Domain Robust Bias Expansion Network (BENet) to enhance face forgery detection. BENet employs an auto-encoder to reconstruct input faces, maintaining the invariance of real faces while selectively enhancing the difference between reconstructed fake faces and their original counterparts. In addition, BENet incorporates a cross-domain detector with a threshold to determine whether the sample belongs to a known distribution.
  arXiv  Detail & Related papers  (2023-10-08T11:30:22Z)
• MD-CSDNetwork: Multi-Domain Cross Stitched Network for Deepfake Detection [80.83725644958633]
  Current deepfake generation methods leave discriminative artifacts in the frequency spectrum of fake images and videos. We present a novel approach, termed as MD-CSDNetwork, for combining the features in the spatial and frequency domains to mine a shared discriminative representation.
  arXiv  Detail & Related papers  (2021-09-15T14:11:53Z)
• A U-Net Based Discriminator for Generative Adversarial Networks [86.67102929147592]
  We propose an alternative U-Net based discriminator architecture for generative adversarial networks (GANs) The proposed architecture allows to provide detailed per-pixel feedback to the generator while maintaining the global coherence of synthesized images. The novel discriminator improves over the state of the art in terms of the standard distribution and image quality metrics.
  arXiv  Detail & Related papers  (2020-02-28T11:16:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.