Towards multi-modal forgery representation learning for AI-generated video detection and localization

• URL: http://arxiv.org/abs/2605.07232v1
• Date: Fri, 08 May 2026 04:35:50 GMT
• Title: Towards multi-modal forgery representation learning for AI-generated video detection and localization
• Abstract summary: AI-generated videos pose escalating risks of semantic distortion and misuse.<n>Most existing AI-generated video detectors remain limited by single- or partial-modality of data modeling.<n>We introduce a core architecture that integrates an LMM semantic branch with atemporal video (ST) visual branch and a multi-scale partial-spoof (PS) audio branch.<n>This multi-modal approach enables simultaneous detection and fine-grained temporal localization of partially manipulated AI-generated video forgeries.
• Score: 8.396379868583718
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Recent advances in generative AI have democratized video creation at scale. AI-generated videos, including partially manipulated clips across visual and audio channels, pose escalating risks of semantic distortion and misuse, which motivates the need for reliable detection tools. Most existing AI-generated video detectors remain limited by single- or partial-modality of data modeling and the lack of fine-grained temporal forgery localization. To address these challenges, our primary novelty introduces a core architecture that jointly integrates an LMM semantic branch with a spatio-temporal (ST) visual branch and a multi-scale partial-spoof (PS) audio branch. This multi-modal approach enables simultaneous detection and fine-grained temporal localization of partially manipulated AI-generated video forgeries. Extensive experiments show that this approach outperforms existing state-of-the-art methods.

Related papers

• Explainable Forensics of Manipulated Segments in Untrimmed Long Videos [50.190474724159465]
  We formulate the task of Temporal AI-Generated Segment localization and Explanation.<n>We introduce TASLE, a large-scale benchmark comprising 12,472 untrimmed videos with diverse manipulation patterns and rich annotation signals.<n>We propose MSLoc, a coarse-to-fine forensic baseline that combines a boundary-sensitive proposal generation module for efficient long-video scanning with an MLLM-based refinement module for precise boundary localization and interpretable reasoning.
  arXiv  Detail & Related papers  (2026-06-01T15:48:38Z)
• Turing Patterns for Multimedia: Reaction-Diffusion Multi-Modal Fusion for Language-Guided Video Moment Retrieval [59.55233913340214]
  Video-language models often struggle to capture dynamic, non-linear interactions between temporal video sequences and textual semantics.<n>We propose textbfReaction-Diffusion Multimodal Fusion (RDMF), a novel framework that reimagines video-language alignment as a reaction-diffusion process.<n>Preliminary experiments demonstrate its potential to outperform existing methods in identifying salient video moments.
  arXiv  Detail & Related papers  (2026-06-01T03:02:35Z)
• CMTA: Leveraging Cross-Modal Temporal Artifacts for Generalizable AI-Generated Video Detection [45.739302264021795]
  Video synthesis techniques pose an unprecedented challenge to digital authenticity.<n>We identify a distinctive fingerprint in cross-modal temporal artifact (CMTA)<n>We propose the CMTA framework, a cross-modal detection approach that captures these unique temporal artifacts.
  arXiv  Detail & Related papers  (2026-05-01T13:04:14Z)
• ATSS: Detecting AI-Generated Videos via Anomalous Temporal Self-Similarity [44.13741922118129]
  We identify a distinctive fingerprint in AIGVs, termed anomalous temporal self-similarity (ATSS)<n>We propose ATSS, a multimodal detection framework that exploits this insight via a triple-similarity representation and a cross-attentive fusion mechanism.<n>ATSS significantly outperforms state-of-the-art methods in terms of AP, AUC, and ACC metrics.
  arXiv  Detail & Related papers  (2026-04-05T09:10:13Z)
• Consolidating Diffusion-Generated Video Detection with Unified Multimodal Forgery Learning [61.3737746844896]
  Existing methods primarily focus on image-level forgery detection, leaving generic video-level forgery detection largely underexplored.<n>We propose a consolidated multimodal detection, named MM-Det++, specifically designed for detecting diffusion-generated videos.
  arXiv  Detail & Related papers  (2025-11-22T16:05:12Z)
• Complementary and Contrastive Learning for Audio-Visual Segmentation [74.11434759171199]
  We present Complementary and Contrastive Transformer (CCFormer), a novel framework adept at processing both local and global information.<n>Our method sets new state-of-the-art benchmarks across the S4, MS3 and AVSS datasets.
  arXiv  Detail & Related papers  (2025-10-11T06:36:59Z)
• BrokenVideos: A Benchmark Dataset for Fine-Grained Artifact Localization in AI-Generated Videos [63.03271511550633]
  BrokenVideos is a benchmark dataset of 3,254 AI-generated videos with meticulously annotated, pixel-level masks highlighting regions of visual corruption.<n>Our experiments show that training state of the art artifact detection models and multi modal large language models (MLLMs) on BrokenVideos significantly improves their ability to localize corrupted regions.
  arXiv  Detail & Related papers  (2025-06-25T03:30:04Z)
• DAVID-XR1: Detecting AI-Generated Videos with Explainable Reasoning [58.70446237944036]
  DAVID-X is the first dataset to pair AI-generated videos with detailed defect-level, temporal-spatial annotations and written rationales.<n>We present DAVID-XR1, a video-language model designed to deliver an interpretable chain of visual reasoning.<n>Our results highlight the promise of explainable detection methods for trustworthy identification of AI-generated video content.
  arXiv  Detail & Related papers  (2025-06-13T13:39:53Z)
• BusterX: MLLM-Powered AI-Generated Video Forgery Detection and Explanation [77.55074597806035]
  GenBuster-200K is a large-scale, high-quality AI-generated video dataset featuring 200K high-resolution video clips.<n>BusterX is a novel AI-generated video detection and explanation framework leveraging multimodal large language model (MLLM) and reinforcement learning.
  arXiv  Detail & Related papers  (2025-05-19T02:06:43Z)
• Exposing AI-generated Videos: A Benchmark Dataset and a Local-and-Global Temporal Defect Based Detection Method [31.763312726582217]
  generative model has made significant advancements in the creation of realistic videos, which causes security issues. In this paper, we first construct a video dataset using advanced diffusion-based video generation algorithms with various semantic contents. By analyzing local and global temporal defects of current AI-generated videos, a novel detection framework is constructed to expose fake videos.
  arXiv  Detail & Related papers  (2024-05-07T09:00:09Z)
• AI-Generated Video Detection via Spatio-Temporal Anomaly Learning [2.1210527985139227]
  Users can easily create non-existent videos to spread false information. A large-scale generated video dataset (GVD) is constructed as a benchmark for model training and evaluation.
  arXiv  Detail & Related papers  (2024-03-25T11:26:18Z)
• AV-Lip-Sync+: Leveraging AV-HuBERT to Exploit Multimodal Inconsistency for Video Deepfake Detection [32.502184301996216]
  Multimodal manipulations (also known as audio-visual deepfakes) make it difficult for unimodal deepfake detectors to detect forgeries in multimedia content. Previous methods mainly adopt uni-modal video forensics and use supervised pre-training for forgery detection. This study proposes a new method based on a multi-modal self-supervised-learning (SSL) feature extractor.
  arXiv  Detail & Related papers  (2023-11-05T18:35:03Z)
• Single-Layer Vision Transformers for More Accurate Early Exits with Less Overhead [88.17413955380262]
  We introduce a novel architecture for early exiting based on the vision transformer architecture. We show that our method works for both classification and regression problems. We also introduce a novel method for integrating audio and visual modalities within early exits in audiovisual data analysis.
  arXiv  Detail & Related papers  (2021-05-19T13:30:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.