JavisDiT: Joint Audio-Video Diffusion Transformer with Hierarchical Spatio-Temporal Prior Synchronization

• URL: http://arxiv.org/abs/2503.23377v1
• Date: Sun, 30 Mar 2025 09:40:42 GMT
• Title: JavisDiT: Joint Audio-Video Diffusion Transformer with Hierarchical Spatio-Temporal Prior Synchronization
• Authors: Kai Liu, Wei Li, Lai Chen, Shengqiong Wu, Yanhao Zheng, Jiayi Ji, Fan Zhou, Rongxin Jiang, Jiebo Luo, Hao Fei, Tat-Seng Chua,
• Abstract summary: JavisDiT is able to generate high-quality audio and video content simultaneously from open-ended user prompts.<n>New benchmark, JavisBench, consists of 10,140 high-quality text-captioned sounding videos spanning diverse scenes and complex real-world scenarios.
• Score: 94.82127738291749
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: This paper introduces JavisDiT, a novel Joint Audio-Video Diffusion Transformer designed for synchronized audio-video generation (JAVG). Built upon the powerful Diffusion Transformer (DiT) architecture, JavisDiT is able to generate high-quality audio and video content simultaneously from open-ended user prompts. To ensure optimal synchronization, we introduce a fine-grained spatio-temporal alignment mechanism through a Hierarchical Spatial-Temporal Synchronized Prior (HiST-Sypo) Estimator. This module extracts both global and fine-grained spatio-temporal priors, guiding the synchronization between the visual and auditory components. Furthermore, we propose a new benchmark, JavisBench, consisting of 10,140 high-quality text-captioned sounding videos spanning diverse scenes and complex real-world scenarios. Further, we specifically devise a robust metric for evaluating the synchronization between generated audio-video pairs in real-world complex content. Experimental results demonstrate that JavisDiT significantly outperforms existing methods by ensuring both high-quality generation and precise synchronization, setting a new standard for JAVG tasks. Our code, model, and dataset will be made publicly available at https://javisdit.github.io/.

Related papers

• Zero-Shot Audio-Visual Editing via Cross-Modal Delta Denoising [114.39028517171236]
  We introduce zero-shot audio-video editing, a novel task that requires transforming original audio-visual content to align with a specified textual prompt without additional model training.<n>To evaluate this task, we curate a benchmark dataset, AvED-Bench, designed explicitly for zero-shot audio-video editing.<n>AvED demonstrates superior results on both AvED-Bench and the recent OAVE dataset to validate its generalization capabilities.
  arXiv  Detail & Related papers  (2025-03-26T17:59:04Z)
• Dual Audio-Centric Modality Coupling for Talking Head Generation [4.03322932416974]
  The generation of audio-driven talking head videos is a key challenge in computer vision and graphics, with applications in virtual avatars and digital media.<n>Traditional approaches often struggle with capturing the complex interaction between audio and facial dynamics, leading to lip synchronization and visual quality issues.<n>We propose a novel NeRF-based framework, Dual Audio-Centric Modality Coupling (DAMC), which effectively integrates content and dynamic features from audio inputs.
  arXiv  Detail & Related papers  (2025-03-26T06:46:51Z)
• SyncVIS: Synchronized Video Instance Segmentation [48.75470418596875]
  We propose to conduct synchronized modeling via a new framework named SyncVIS.<n>SyncVIS explicitly introduces video-level query embeddings and designs two key modules to synchronize video-level query with frame-level query embeddings.<n>The proposed approach achieves state-of-the-art results, which demonstrates the effectiveness and generality of the proposed approach.
  arXiv  Detail & Related papers  (2024-12-01T16:43:20Z)
• Masked Generative Video-to-Audio Transformers with Enhanced Synchronicity [12.848371604063168]
  We propose a V2A generative model, named MaskVAT, that interconnects a full-band high-quality general audio with a sequence-to-sequence masked generative model. Our results show that, by combining a high-quality with the proper pre-trained audio-visual features and a sequence-to-sequence parallel structure, we are able to yield highly synchronized results.
  arXiv  Detail & Related papers  (2024-07-15T01:49:59Z)
• Text-to-Audio Generation Synchronized with Videos [44.848393652233796]
  We introduce a groundbreaking benchmark for Text-to-Audio generation that aligns with Videos, named T2AV-Bench. We also present a simple yet effective video-aligned TTA generation model, namely T2AV. It employs a temporal multi-head attention transformer to extract and understand temporal nuances from video data, a feat amplified by our Audio-Visual ControlNet.
  arXiv  Detail & Related papers  (2024-03-08T22:27:38Z)
• Synchformer: Efficient Synchronization from Sparse Cues [100.89656994681934]
  Our contributions include a novel audio-visual synchronization model, and training that decouples extraction from synchronization modelling. This approach achieves state-of-the-art performance in both dense and sparse settings. We also extend synchronization model training to AudioSet a million-scale 'in-the-wild' dataset, investigate evidence attribution techniques for interpretability, and explore a new capability for synchronization models: audio-visual synchronizability.
  arXiv  Detail & Related papers  (2024-01-29T18:59:55Z)
• Sparse in Space and Time: Audio-visual Synchronisation with Trainable Selectors [103.21152156339484]
  The objective of this paper is audio-visual synchronisation of general videos 'in the wild' We make four contributions: (i) in order to handle longer temporal sequences required for sparse synchronisation signals, we design a multi-modal transformer model that employs'selectors' We identify artefacts that can arise from the compression codecs used for audio and video and can be used by audio-visual models in training to artificially solve the synchronisation task.
  arXiv  Detail & Related papers  (2022-10-13T14:25:37Z)
• Audio-Visual Synchronisation in the wild [149.84890978170174]
  We identify and curate a test set with high audio-visual correlation, namely VGG-Sound Sync. We compare a number of transformer-based architectural variants specifically designed to model audio and visual signals of arbitrary length. We set the first benchmark for general audio-visual synchronisation with over 160 diverse classes in the new VGG-Sound Sync video dataset.
  arXiv  Detail & Related papers  (2021-12-08T17:50:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.