SAVe: Self-Supervised Audio-visual Deepfake Detection Exploiting Visual Artifacts and Audio-visual Misalignment

• URL: http://arxiv.org/abs/2603.25140v1
• Date: Thu, 26 Mar 2026 08:01:35 GMT
• Title: SAVe: Self-Supervised Audio-visual Deepfake Detection Exploiting Visual Artifacts and Audio-visual Misalignment
• Abstract summary: We propose SAVe, a self-supervised audio-visual deepfake detection framework that learns entirely on authentic videos.<n>SAVE generates on-the-fly, identity-preserving, region-aware self-blended pseudo-manipulations to emulate tampering artifacts.<n>To capture cross-modal evidence, SAVe also models lip-speech synchronization.
• Score: 74.05278327933006
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal deepfakes can exhibit subtle visual artifacts and cross-modal inconsistencies, which remain challenging to detect, especially when detectors are trained primarily on curated synthetic forgeries. Such synthetic dependence can introduce dataset and generator bias, limiting scalability and robustness to unseen manipulations. We propose SAVe, a self-supervised audio-visual deepfake detection framework that learns entirely on authentic videos. SAVe generates on-the-fly, identity-preserving, region-aware self-blended pseudo-manipulations to emulate tampering artifacts, enabling the model to learn complementary visual cues across multiple facial granularities. To capture cross-modal evidence, SAVe also models lip-speech synchronization via an audio-visual alignment component that detects temporal misalignment patterns characteristic of audio-visual forgeries. Experiments on FakeAVCeleb and AV-LipSync-TIMIT demonstrate competitive in-domain performance and strong cross-dataset generalization, highlighting self-supervised learning as a scalable paradigm for multimodal deepfake detection.

Related papers

• From Talking to Singing: A New Challenge for Audio-Visual Deepfake Detection [23.76387977850747]
  Existing methods for audio-visual deepfake detection rely on cross-modal inconsistencies.<n>In singing, rhythmic vocalization weakens this coupling and introduces a nontrivial domain shift.<n>We construct the Singing Head DeepFake dataset using rhythm-aware generative models to fill the gap in singing benchmarks.
  arXiv  Detail & Related papers  (2026-05-27T04:33:39Z)
• Unleashing Vision-Language Semantics for Deepfake Video Detection [78.7562836979696]
  Deepfake Video Detection (DFD) studies have demonstrated that pre-trained Vision-Language Models (VLMs) exhibit strong generalization capabilities in detecting artifacts across different identities.<n>We propose VLAForge, a novel DFD framework that unleashes the potential of such cross-modal semantics to enhance model's discriminability in deepfake detection.
  arXiv  Detail & Related papers  (2026-03-25T16:05:35Z)
• KLASSify to Verify: Audio-Visual Deepfake Detection Using SSL-based Audio and Handcrafted Visual Features [1.488627850405606]
  We propose multimodal approaches for the AV-Deepfake1M 2025 challenge.<n>For the visual modality, we leverage handcrafted features to improve interpretability and adaptability.<n>For the audio modality, we adapt a self-supervised learning backbone coupled with graph attention networks to capture rich audio representations.<n>Our approach strikes a balance between performance and real-world deployment, focusing on resilience and potential interpretability.
  arXiv  Detail & Related papers  (2025-08-10T13:29:08Z)
• AVadCLIP: Audio-Visual Collaboration for Robust Video Anomaly Detection [57.649223695021114]
  We present a novel weakly supervised framework that leverages audio-visual collaboration for robust video anomaly detection.<n>Our framework demonstrates superior performance across multiple benchmarks, with audio integration significantly boosting anomaly detection accuracy.
  arXiv  Detail & Related papers  (2025-04-06T13:59:16Z)
• Contextual Cross-Modal Attention for Audio-Visual Deepfake Detection and Localization [3.9440964696313485]
  In the digital age, the emergence of deepfakes and synthetic media presents a significant threat to societal and political integrity. Deepfakes based on multi-modal manipulation, such as audio-visual, are more realistic and pose a greater threat. We propose a novel multi-modal attention framework based on recurrent neural networks (RNNs) that leverages contextual information for audio-visual deepfake detection.
  arXiv  Detail & Related papers  (2024-08-02T18:45:01Z)
• AV-Lip-Sync+: Leveraging AV-HuBERT to Exploit Multimodal Inconsistency for Video Deepfake Detection [32.502184301996216]
  Multimodal manipulations (also known as audio-visual deepfakes) make it difficult for unimodal deepfake detectors to detect forgeries in multimedia content. Previous methods mainly adopt uni-modal video forensics and use supervised pre-training for forgery detection. This study proposes a new method based on a multi-modal self-supervised-learning (SSL) feature extractor.
  arXiv  Detail & Related papers  (2023-11-05T18:35:03Z)
• AVTENet: A Human-Cognition-Inspired Audio-Visual Transformer-Based Ensemble Network for Video Deepfake Detection [49.81915942821647]
  This study introduces the audio-visual transformer-based ensemble network (AVTENet) to detect deepfake videos.<n>For evaluation, we use the recently released benchmark multimodal audio-video FakeAVCeleb dataset.<n>For a detailed analysis, we evaluate AVTENet, its variants, and several existing methods on multiple test sets of the FakeAVCeleb dataset.
  arXiv  Detail & Related papers  (2023-10-19T19:01:26Z)
• MIS-AVoiDD: Modality Invariant and Specific Representation for Audio-Visual Deepfake Detection [4.659427498118277]
  A novel kind of deepfakes has emerged with either audio or visual modalities manipulated. Existing multimodal deepfake detectors are often based on the fusion of the audio and visual streams from the video. In this paper, we tackle the problem at the representation level to aid the fusion of audio and visual streams for multimodal deepfake detection.
  arXiv  Detail & Related papers  (2023-10-03T17:43:24Z)
• NPVForensics: Jointing Non-critical Phonemes and Visemes for Deepfake Detection [50.33525966541906]
  Existing multimodal detection methods capture audio-visual inconsistencies to expose Deepfake videos. We propose a novel Deepfake detection method to mine the correlation between Non-critical Phonemes and Visemes, termed NPVForensics. Our model can be easily adapted to the downstream Deepfake datasets with fine-tuning.
  arXiv  Detail & Related papers  (2023-06-12T06:06:05Z)
• Beyond the Spectrum: Detecting Deepfakes via Re-Synthesis [69.09526348527203]
  Deep generative models have led to highly realistic media, known as deepfakes, that are commonly indistinguishable from real to human eyes. We propose a novel fake detection that is designed to re-synthesize testing images and extract visual cues for detection. We demonstrate the improved effectiveness, cross-GAN generalization, and robustness against perturbations of our approach in a variety of detection scenarios.
  arXiv  Detail & Related papers  (2021-05-29T21:22:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.