From Talking to Singing: A New Challenge for Audio-Visual Deepfake Detection

• URL: http://arxiv.org/abs/2605.27944v1
• Date: Wed, 27 May 2026 04:33:39 GMT
• Title: From Talking to Singing: A New Challenge for Audio-Visual Deepfake Detection
• Abstract summary: Existing methods for audio-visual deepfake detection rely on cross-modal inconsistencies.<n>In singing, rhythmic vocalization weakens this coupling and introduces a nontrivial domain shift.<n>We construct the Singing Head DeepFake dataset using rhythm-aware generative models to fill the gap in singing benchmarks.
• Score: 23.76387977850747
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: With rapid advances in audio-visual generative models, reliable forgery detection becomes increasingly critical. Existing methods for audio-visual deepfake detection typically rely on cross-modal inconsistencies. In singing, rhythmic vocalization weakens this coupling and introduces a nontrivial domain shift, substantially degrading detection performance. We construct the Singing Head DeepFake (SHDF) dataset using rhythm-aware generative models to fill the gap in singing benchmarks. To cope with cross-scenario domain shifts, we propose a Text-guided Audio-Visual Forgery Detection (T-AVFD) framework that generalizes across both talking and singing scenarios. T-AVFD comprises a facial authenticity pattern learner and a multi-modal differential weight learning module. The pattern learner aligns facial features with multi-granularity textual descriptions to learn generalizable authenticity patterns. The weight learning module preserves intrinsic audio-visual consistency and adaptively integrates it with authenticity patterns via differential weighting. Extensive experiments on multiple talking head deepfake datasets and SHDF show consistent improvements over existing baselines and strong robustness under diverse perturbations.

Related papers

• SAVe: Self-Supervised Audio-visual Deepfake Detection Exploiting Visual Artifacts and Audio-visual Misalignment [74.05278327933006]
  We propose SAVe, a self-supervised audio-visual deepfake detection framework that learns entirely on authentic videos.<n>SAVE generates on-the-fly, identity-preserving, region-aware self-blended pseudo-manipulations to emulate tampering artifacts.<n>To capture cross-modal evidence, SAVe also models lip-speech synchronization.
  arXiv  Detail & Related papers  (2026-03-26T08:01:35Z)
• Towards Generalizable Deepfake Detection via Forgery-aware Audio-Visual Adaptation: A Variational Bayesian Approach [31.10567291555587]
  Forgery-aware Audio-Visual Adaptation with Variational Bayes (FoVB) is developed.<n>We exploit various difference convolutions and a high-pass filter to discern local and global forgery traces from both modalities.<n>Our FoVB outperforms other state-of-the-art methods in various benchmarks.
  arXiv  Detail & Related papers  (2025-11-24T13:20:03Z)
• PIA: Deepfake Detection Using Phoneme-Temporal and Identity-Dynamic Analysis [22.49571926071395]
  Phoneme-Temporal and Identity-Dynamic Analysis is a novel framework for deepfake detection.<n>It incorporates language, dynamic face motion, and facial identification cues to address limitations.<n>It significantly improves the detection of subtle deepfake alterations.
  arXiv  Detail & Related papers  (2025-10-16T02:51:42Z)
• Learning Normal Patterns in Musical Loops [0.0]
  This paper introduces an unsupervised framework for detecting audio patterns in musical samples (loops) through anomaly detection techniques.<n>We address these limitations through an architecture combining deep feature extraction with unsupervised anomaly detection.
  arXiv  Detail & Related papers  (2025-05-22T19:52:00Z)
• AVadCLIP: Audio-Visual Collaboration for Robust Video Anomaly Detection [57.649223695021114]
  We present a novel weakly supervised framework that leverages audio-visual collaboration for robust video anomaly detection.<n>Our framework demonstrates superior performance across multiple benchmarks, with audio integration significantly boosting anomaly detection accuracy.
  arXiv  Detail & Related papers  (2025-04-06T13:59:16Z)
• DiMoDif: Discourse Modality-information Differentiation for Audio-visual Deepfake Detection and Localization [13.840950434728533]
  DiMoDif is an audio-visual deepfake detection framework.<n>It exploits the inter-modality differences in machine perception of speech.<n>It temporally localizes the deepfake forgery.
  arXiv  Detail & Related papers  (2024-11-15T13:47:33Z)
• Training-Free Deepfake Voice Recognition by Leveraging Large-Scale Pre-Trained Models [52.04189118767758]
  Generalization is a main issue for current audio deepfake detectors. In this paper we study the potential of large-scale pre-trained models for audio deepfake detection.
  arXiv  Detail & Related papers  (2024-05-03T15:27:11Z)
• What to Remember: Self-Adaptive Continual Learning for Audio Deepfake Detection [53.063161380423715]
  Existing detection models have shown remarkable success in discriminating known deepfake audio, but struggle when encountering new attack types. We propose a continual learning approach called Radian Weight Modification (RWM) for audio deepfake detection.
  arXiv  Detail & Related papers  (2023-12-15T09:52:17Z)
• AVTENet: A Human-Cognition-Inspired Audio-Visual Transformer-Based Ensemble Network for Video Deepfake Detection [49.81915942821647]
  This study introduces the audio-visual transformer-based ensemble network (AVTENet) to detect deepfake videos.<n>For evaluation, we use the recently released benchmark multimodal audio-video FakeAVCeleb dataset.<n>For a detailed analysis, we evaluate AVTENet, its variants, and several existing methods on multiple test sets of the FakeAVCeleb dataset.
  arXiv  Detail & Related papers  (2023-10-19T19:01:26Z)
• Anomalous Sound Detection using Audio Representation with Machine ID based Contrastive Learning Pretraining [52.191658157204856]
  This paper uses contrastive learning to refine audio representations for each machine ID, rather than for each audio sample. The proposed two-stage method uses contrastive learning to pretrain the audio representation model. Experiments show that our method outperforms the state-of-the-art methods using contrastive learning or self-supervised classification.
  arXiv  Detail & Related papers  (2023-04-07T11:08:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.