Deepfake News Detection: A Multimodal Framework Integrating LipNet, DeepSpeech and ResNET for Enhanced Audio-Visual Analysis

• URL: http://arxiv.org/abs/2607.20579v1
• Date: Wed, 22 Jul 2026 11:39:53 GMT
• Title: Deepfake News Detection: A Multimodal Framework Integrating LipNet, DeepSpeech and ResNET for Enhanced Audio-Visual Analysis
• Abstract summary: Deepfake news refers to AI-generated (or AI ma-nipulated) multimedia content intentionally generated to deceive audiences.<n>This paper presents a framework that discerns the authenticity of video content by jointly exploiting audio and visual cues.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Deepfake news refers to AI-generated (or AI ma-nipulated) multimedia content intentionally generated to deceive audiences by manipulating the facial expressions, or speech while maintaining the realistic appearance. The rapid progress of generative AI has made the synthesis of highly realistic fake videos and cloned voices widely accessible, posing a serious threat to the authenticity of digital news media. This paper presents a multi-modal framework that discerns the authenticity of video content by jointly exploiting audio and visual cues, thereby addressing the challenge of detecting the deepfake videos. We proposed a framework that involves features extraction from lip movements, audio content and video frames. Lip movements and speech content are encoded using the LipNet and DeepSpeech2 models, while facial features are extracted by leveraging the use of BlazeFace and represented with ResNet18. The extracted feature vectors are concatenated into a holistic video representation and classified with an ensemble of machine learning and deep learning models, including Random Forest (RF), Multi-layer Perceptron (MLP) and Long Short-Term Memory (LSTM) networks. Exten-sive experiments performed on the FakeAVCeleb dataset shows that the proposed approach attains an accuracy of 94% using augmented audio features, outperforming a state-of-the-art multi-modal ensemble baseline. The results confirm the robustness and practical potential of the proposed framework for deepfake news detection.

Related papers

• MAVIN: Multi-Shot Audio-Visual Generation with Narrative Control [66.04301887685004]
  Existing methods suffer from temporal misalignment, limited controllability, and incomplete scripting.<n>We propose MAVIN, the first framework for multi-shot audio-visual generation with customized narrative control.<n> MAVIN achieves state-of-the-art performance, opening up a new avenue for integrating generative models into professional filmmaking.
  arXiv  Detail & Related papers  (2026-06-28T16:01:04Z)
• Explainable Forensics of Manipulated Segments in Untrimmed Long Videos [50.190474724159465]
  We formulate the task of Temporal AI-Generated Segment localization and Explanation.<n>We introduce TASLE, a large-scale benchmark comprising 12,472 untrimmed videos with diverse manipulation patterns and rich annotation signals.<n>We propose MSLoc, a coarse-to-fine forensic baseline that combines a boundary-sensitive proposal generation module for efficient long-video scanning with an MLLM-based refinement module for precise boundary localization and interpretable reasoning.
  arXiv  Detail & Related papers  (2026-06-01T15:48:38Z)
• EDVD-LLaMA: Explainable Deepfake Video Detection via Multimodal Large Language Model Reasoning [58.42596067220998]
  deepfake video technology has not only facilitated artistic creation but also made it easier to spread misinformation.<n>Traditional deepfake video detection methods face issues such as a lack of transparency in their principles and insufficient capabilities to cope with forgery techniques.<n>This paper proposes the explainable deepfake video detection (EDVD) task and designs the EDVD-LLaMA multimodal reasoning framework.
  arXiv  Detail & Related papers  (2025-10-18T10:34:05Z)
• Lightweight Joint Audio-Visual Deepfake Detection via Single-Stream Multi-Modal Learning Framework [19.53717894228692]
  Deepfakes are AI-synthesized multimedia data that may be abused for spreading misinformation.<n>We propose a lightweight network for audio-visual deepfake detection via a single-stream multi-modal learning framework.<n>Our method is significantly lightweight with only 0.48M parameters, yet it achieves superiority in both uni-modal and multi-modal deepfakes.
  arXiv  Detail & Related papers  (2025-06-09T02:13:04Z)
• A Multimodal Framework for Deepfake Detection [0.0]
  Deepfakes, synthetic media created using AI, can convincingly alter videos and audio to misrepresent reality. Our research addresses the critical issue of deepfakes through an innovative multimodal approach. Our framework combines visual and auditory analyses, yielding an accuracy of 94%.
  arXiv  Detail & Related papers  (2024-10-04T14:59:10Z)
• Vulnerability of Automatic Identity Recognition to Audio-Visual Deepfakes [13.042731289687918]
  We present the first realistic audio-visual database of deepfakes SWAN-DF, where lips and speech are well synchronized. We demonstrate the vulnerability of a state of the art speaker recognition system, such as ECAPA-TDNN-based model from SpeechBrain.
  arXiv  Detail & Related papers  (2023-11-29T14:18:04Z)
• AVTENet: A Human-Cognition-Inspired Audio-Visual Transformer-Based Ensemble Network for Video Deepfake Detection [49.81915942821647]
  This study introduces the audio-visual transformer-based ensemble network (AVTENet) to detect deepfake videos.<n>For evaluation, we use the recently released benchmark multimodal audio-video FakeAVCeleb dataset.<n>For a detailed analysis, we evaluate AVTENet, its variants, and several existing methods on multiple test sets of the FakeAVCeleb dataset.
  arXiv  Detail & Related papers  (2023-10-19T19:01:26Z)
• DF-TransFusion: Multimodal Deepfake Detection via Lip-Audio Cross-Attention and Facial Self-Attention [13.671150394943684]
  We present a novel multi-modal audio-video framework designed to concurrently process audio and video inputs for deepfake detection tasks. Our model capitalizes on lip synchronization with input audio through a cross-attention mechanism while extracting visual cues via a fine-tuned VGG-16 network.
  arXiv  Detail & Related papers  (2023-09-12T18:37:05Z)
• Audio-Visual Person-of-Interest DeepFake Detection [77.04789677645682]
  The aim of this work is to propose a deepfake detector that can cope with the wide variety of manipulation methods and scenarios encountered in the real world. We leverage a contrastive learning paradigm to learn the moving-face and audio segment embeddings that are most discriminative for each identity. Our method can detect both single-modality (audio-only, video-only) and multi-modality (audio-video) attacks, and is robust to low-quality or corrupted videos.
  arXiv  Detail & Related papers  (2022-04-06T20:51:40Z)
• Learning to Predict Salient Faces: A Novel Visual-Audio Saliency Model [96.24038430433885]
  We propose a novel multi-modal video saliency model consisting of three branches: visual, audio and face. Experimental results show that the proposed method outperforms 11 state-of-the-art saliency prediction works.
  arXiv  Detail & Related papers  (2021-03-29T09:09:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.