CAD: A General Multimodal Framework for Video Deepfake Detection via Cross-Modal Alignment and Distillation

• URL: http://arxiv.org/abs/2505.15233v1
• Date: Wed, 21 May 2025 08:11:07 GMT
• Title: CAD: A General Multimodal Framework for Video Deepfake Detection via Cross-Modal Alignment and Distillation
• Abstract summary: We propose a general framework for video deepfake detection via Cross-Modal Alignment and Distillation (CAD)<n>CAD comprises two core components: 1) Cross-modal alignment that identifies inconsistencies in high-level semantic synchronization (e.g., lip-speech mismatches); 2) Cross-modal distillation that mitigates mismatchs while preserving modality-specific forensic traces (e.g., spectral distortions in synthetic audio)
• Score: 24.952907733127223
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: The rapid emergence of multimodal deepfakes (visual and auditory content are manipulated in concert) undermines the reliability of existing detectors that rely solely on modality-specific artifacts or cross-modal inconsistencies. In this work, we first demonstrate that modality-specific forensic traces (e.g., face-swap artifacts or spectral distortions) and modality-shared semantic misalignments (e.g., lip-speech asynchrony) offer complementary evidence, and that neglecting either aspect limits detection performance. Existing approaches either naively fuse modality-specific features without reconciling their conflicting characteristics or focus predominantly on semantic misalignment at the expense of modality-specific fine-grained artifact cues. To address these shortcomings, we propose a general multimodal framework for video deepfake detection via Cross-Modal Alignment and Distillation (CAD). CAD comprises two core components: 1) Cross-modal alignment that identifies inconsistencies in high-level semantic synchronization (e.g., lip-speech mismatches); 2) Cross-modal distillation that mitigates feature conflicts during fusion while preserving modality-specific forensic traces (e.g., spectral distortions in synthetic audio). Extensive experiments on both multimodal and unimodal (e.g., image-only/video-only)deepfake benchmarks demonstrate that CAD significantly outperforms previous methods, validating the necessity of harmonious integration of multimodal complementary information.

Related papers

• XMatchAD: A Cross-Modal Matching Perspective on Reconstruction-based Anomaly Detection [30.401044523780723]
  XMatchAD is a novel UAD framework that reinterprets the task from a pseudo cross-modal matching perspective.<n>Our method consistently achieves superior performance, outperforming state-of-the-art methods in multi-class anomaly detection and localization.
  arXiv  Detail & Related papers  (2026-07-26T13:47:10Z)
• Attribution-Guided Multimodal Deepfake Detection via Cross-Modal Forensic Fingerprints [4.598350938853635]
  We argue that a detector incapable of identifying how a video was forged is likely learning the wrong signal.<n>Unlike binary detection, attribution-guided learning imposes a stronger geometric constraint on the shared embedding space.<n>We propose the Attribution-Guided Multimodal Deepfake Detection framework, which jointly learns to detect and attribute manipulation.
  arXiv  Detail & Related papers  (2026-04-29T09:11:13Z)
• MSCT: Differential Cross-Modal Attention for Deepfake Detection [8.183317636944706]
  deepfake detection typically employs a complementary multi-modal model to check the forgery traces in the video.<n>Traditional multi-modal forgery detection method has the problem of insufficient feature extraction and modal alignment deviation.<n>Our approach includes a multi-scale self-attention to integrate the features of adjacent embeddings and a differential cross-modal attention to fuse multi-modal features.
  arXiv  Detail & Related papers  (2026-04-09T02:56:16Z)
• SAVe: Self-Supervised Audio-visual Deepfake Detection Exploiting Visual Artifacts and Audio-visual Misalignment [74.05278327933006]
  We propose SAVe, a self-supervised audio-visual deepfake detection framework that learns entirely on authentic videos.<n>SAVE generates on-the-fly, identity-preserving, region-aware self-blended pseudo-manipulations to emulate tampering artifacts.<n>To capture cross-modal evidence, SAVe also models lip-speech synchronization.
  arXiv  Detail & Related papers  (2026-03-26T08:01:35Z)
• Complementarity-Supervised Spectral-Band Routing for Multimodal Emotion Recognition [60.20529806857076]
  Multimodal emotion recognition fuses cues such as text, video, and audio to understand individual emotional states.<n>Prior methods face two main limitations: mechanically relying on independent unimodal performance, and coarse-grained fusion conflicting with the fine-grained representations required by emotion tasks.<n>We propose the Complementarity-Supervised Multi-Band Expert Network, named Atsuko, to model fine-grained complementary features via multi-scale band decomposition and expert collaboration.
  arXiv  Detail & Related papers  (2026-03-07T03:58:48Z)
• Test-time Adaptive Hierarchical Co-enhanced Denoising Network for Reliable Multimodal Classification [55.56234913868664]
  We propose Test-time Adaptive Hierarchical Co-enhanced Denoising Network (TAHCD) for reliable learning on multimodal data.<n>The proposed method achieves superior classification performance, robustness, and generalization compared with state-of-the-art reliable multimodal learning approaches.
  arXiv  Detail & Related papers  (2026-01-12T03:14:12Z)
• CLASH: A Benchmark for Cross-Modal Contradiction Detection [15.134491772506196]
  CLASH is a novel benchmark for multimodal contradiction detection.<n>It features COCO images paired with contradictory captions containing controlled object-level or attribute-level contradictions.
  arXiv  Detail & Related papers  (2025-11-24T15:09:07Z)
• CrossCheck-Bench: Diagnosing Compositional Failures in Multimodal Conflict Resolution [20.823419395675412]
  CrossCheck-Bench is a diagnostic benchmark for evaluating contradiction detection in multimodal inputs.<n>We evaluate 13 state-of-the-art vision-language models and observe a consistent performance drop as tasks shift from perceptual matching to logical contradiction detection.
  arXiv  Detail & Related papers  (2025-11-19T12:17:15Z)
• Double Helix Diffusion for Cross-Domain Anomaly Image Generation [47.093354259479234]
  This paper introduces Double Helix Diffusion (DH-Diff), a novel cross-domain generative framework designed to simultaneously synthesize high-fidelity anomaly images and their pixel-level annotation masks.<n>DH-Diff employs a unique architecture inspired by a double helix, cycling through distinct modules for feature separation, connection, and merging.<n>Extensive experiments demonstrate that DH-Diff significantly outperforms state-of-the-art methods in diversity and authenticity, leading to significant improvements in downstream anomaly detection performance.
  arXiv  Detail & Related papers  (2025-09-16T08:06:07Z)
• Beyond Artificial Misalignment: Detecting and Grounding Semantic-Coordinated Multimodal Manipulations [56.816929931908824]
  We pioneer the detection of semantically-coordinated manipulations in multimodal data.<n>We propose a Retrieval-Augmented Manipulation Detection and Grounding (RamDG) framework.<n>Our framework significantly outperforms existing methods, achieving 2.06% higher detection accuracy on SAMM compared to state-of-the-art approaches.
  arXiv  Detail & Related papers  (2025-09-16T04:18:48Z)
• Two Views, One Truth: Spectral and Self-Supervised Features Fusion for Robust Speech Deepfake Detection [11.121265242990166]
  Recent advances in synthetic speech have made audio deepfakes increasingly realistic, posing significant security risks.<n>Existing detection methods that rely on a single modality, either raw waveform embeddings or spectral based features, are vulnerable to non spoof disturbances.<n>We investigate hybrid fusion frameworks that integrate self supervised learning (SSL) based representations with handcrafted spectral descriptors.
  arXiv  Detail & Related papers  (2025-07-27T21:22:27Z)
• METER: Multi-modal Evidence-based Thinking and Explainable Reasoning -- Algorithm and Benchmark [48.78602579128459]
  We introduce METER, a unified benchmark for interpretable forgery detection spanning images, videos, audio, and audio-visual content.<n>Our dataset comprises four tracks, each requiring not only real-vs-fake classification but also evidence-chain-based explanations.
  arXiv  Detail & Related papers  (2025-07-22T03:42:51Z)
• Multimodal LLM-Guided Semantic Correction in Text-to-Image Diffusion [52.315729095824906]
  MLLM Semantic-Corrected Ping-Pong-Ahead Diffusion (PPAD) is a novel framework that introduces a Multimodal Large Language Model (MLLM) as a semantic observer during inference.<n>It performs real-time analysis on intermediate generations, identifies latent semantic inconsistencies, and translates feedback into controllable signals that actively guide the remaining denoising steps.<n>Extensive experiments demonstrate PPAD's significant improvements.
  arXiv  Detail & Related papers  (2025-05-26T14:42:35Z)
• Aligning First, Then Fusing: A Novel Weakly Supervised Multimodal Violence Detection Method [11.01048485795428]
  We propose a new weakly supervised violence detection framework.<n>It consists of unimodal multiple-instance learning for extracting unimodal semantic features, multimodal alignment, multimodal fusion, and final detection.<n> Experimental results on benchmark datasets demonstrate the effectiveness of our method.
  arXiv  Detail & Related papers  (2025-01-13T17:14:25Z)
• Multi-task Visual Grounding with Coarse-to-Fine Consistency Constraints [15.541287957548771]
  We propose a Coarse-to-fine Consistency Constraints Visual Grounding architecture.<n>It integrates implicit and explicit modeling approaches within a two-stage framework.<n>It significantly outperforms state-of-the-art REC and RIS methods by a substantial margin.
  arXiv  Detail & Related papers  (2025-01-12T04:30:13Z)
• Cross-Modal Learning for Anomaly Detection in Complex Industrial Process: Methodology and Benchmark [19.376814754500625]
  Anomaly detection in complex industrial processes plays a pivotal role in ensuring efficient, stable, and secure operation. This paper proposes a cross-modal Transformer to facilitate anomaly detection by exploring the correlation between visual features (video) and process variables (current) in the context of the fused magnesium smelting process. We present a pioneering cross-modal benchmark of the fused magnesium smelting process, featuring synchronously acquired video and current data for over 2.2 million samples.
  arXiv  Detail & Related papers  (2024-06-13T11:40:06Z)
• Improving Vision Anomaly Detection with the Guidance of Language Modality [64.53005837237754]
  This paper tackles the challenges for vision modality from a multimodal point of view. We propose Cross-modal Guidance (CMG) to tackle the redundant information issue and sparse space issue. To learn a more compact latent space for the vision anomaly detector, CMLE learns a correlation structure matrix from the language modality.
  arXiv  Detail & Related papers  (2023-10-04T13:44:56Z)
• MMNet: Multi-Collaboration and Multi-Supervision Network for Sequential Deepfake Detection [81.59191603867586]
  Sequential deepfake detection aims to identify forged facial regions with the correct sequence for recovery. The recovery of forged images requires knowledge of the manipulation model to implement inverse transformations. We propose Multi-Collaboration and Multi-Supervision Network (MMNet) that handles various spatial scales and sequential permutations in forged face images.
  arXiv  Detail & Related papers  (2023-07-06T02:32:08Z)
• Cross-Attention is Not Enough: Incongruity-Aware Dynamic Hierarchical Fusion for Multimodal Affect Recognition [69.32305810128994]
  Incongruity between modalities poses a challenge for multimodal fusion, especially in affect recognition. We propose the Hierarchical Crossmodal Transformer with Dynamic Modality Gating (HCT-DMG), a lightweight incongruity-aware model. HCT-DMG: 1) outperforms previous multimodal models with a reduced size of approximately 0.8M parameters; 2) recognizes hard samples where incongruity makes affect recognition difficult; 3) mitigates the incongruity at the latent level in crossmodal attention.
  arXiv  Detail & Related papers  (2023-05-23T01:24:15Z)
• Object Segmentation by Mining Cross-Modal Semantics [68.88086621181628]
  We propose a novel approach by mining the Cross-Modal Semantics to guide the fusion and decoding of multimodal features. Specifically, we propose a novel network, termed XMSNet, consisting of (1) all-round attentive fusion (AF), (2) coarse-to-fine decoder (CFD), and (3) cross-layer self-supervision.
  arXiv  Detail & Related papers  (2023-05-17T14:30:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.