Diffusion Models for Joint Audio-Video Generation

• URL: http://arxiv.org/abs/2603.16093v1
• Date: Tue, 17 Mar 2026 03:31:37 GMT
• Title: Diffusion Models for Joint Audio-Video Generation
• Abstract summary: I release two high-quality, paired audio-video datasets.<n>I train the MM-Diffusion architecture from scratch on our datasets.<n>I propose a sequential two-step text-to-audio-video generation pipeline.
• Score: 51.56484100374058
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Multimodal generative models have shown remarkable progress in single-modality video and audio synthesis, yet truly joint audio-video generation remains an open challenge. In this paper, I explore four key contributions to advance this field. First, I release two high-quality, paired audio-video datasets. The datasets consisting on 13 hours of video-game clips and 64 hours of concert performances, each segmented into consistent 34-second samples to facilitate reproducible research. Second, I train the MM-Diffusion architecture from scratch on our datasets, demonstrating its ability to produce semantically coherent audio-video pairs and quantitatively evaluating alignment on rapid actions and musical cues. Third, I investigate joint latent diffusion by leveraging pretrained video and audio encoder-decoders, uncovering challenges and inconsistencies in the multimodal decoding stage. Finally, I propose a sequential two-step text-to-audio-video generation pipeline: first generating video, then conditioning on both the video output and the original prompt to synthesize temporally synchronized audio. My experiments show that this modular approach yields high-fidelity generations of audio video generation.

Related papers

• DreamX-Creator: Democratizing Native Audio-Video Generation at 2K Resolution [16.570672947240855]
  We present DreamX-Creator 1.0, a compact native joint audio-video generation system centered on a 7B generator.<n>A unified Audio-Video Data System constructs and filters temporally coherent clips, produces structured multimodal annotations, and organizes clips into capability-oriented data pools.<n>For high-resolution output, our Autoregressive 1-Step 2K Refinement pipeline adapts a bidirectional multi-step teacher into an autoregressive multi-step refiner.
  arXiv  Detail & Related papers  (2026-08-31T17:11:05Z)
• Encore: Infinite Audio-Video Generation with Adaptive Signal Routing [22.689981874924943]
  Existing audio-video generation methods produce well-synchronized clips but are limited to short durations.<n>Long-video generation methods extend duration through chunk-based iterative synthesis yet lack audio entirely.<n>We present Encore for long-form synchronized audio-video generation.
  arXiv  Detail & Related papers  (2026-08-28T18:10:18Z)
• OmniVR: Joint Video-Audio Conditional Generation for Restoring Degraded Historical Films [77.65090601396003]
  Historical films suffer from co-occurring visual and audio degradations.<n>We present OmniVR, the first joint audio-video generative restoration model.
  arXiv  Detail & Related papers  (2026-08-04T21:00:57Z)
• CineDance: Towards Next-Generation Multi-Shot Long-Form Cinematic Audio-Video Generation [107.75241387090271]
  We introduce CineDance-1M, a large-scale, open research Text-to-Audio-Video dataset.<n>Averaging 92.8 seconds and 24.2 continuous shots per video, it provides, structured annotations for both audio and video modalities.<n>For a comprehensive assessment, we propose CineBench, featuring a diverse prompt suite and a six-dimensional, human-aligned metric system.
  arXiv  Detail & Related papers  (2026-06-08T15:35:51Z)
• HunyuanVideo-Foley: Multimodal Diffusion with Representation Alignment for High-Fidelity Foley Audio Generation [14.921126281071544]
  HunyuanVideo-Foley is an end-to-end text-video-to-audio framework.<n>It synthesizes high-fidelity audio precisely aligned with visual dynamics and semantic context.<n>It achieves new state-of-the-art performance across audio fidelity, visual-semantic alignment, temporal alignment and distribution matching.
  arXiv  Detail & Related papers  (2025-08-23T07:30:18Z)
• Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation [89.96013329530484]
  We consider the task of generating diverse and realistic videos guided by natural audio samples from a wide variety of semantic classes. We utilize an existing text-conditioned video generation model and a pre-trained audio encoder model. We validate our method extensively on three datasets demonstrating significant semantic diversity of audio-video samples.
  arXiv  Detail & Related papers  (2023-09-28T13:26:26Z)
• Large-scale unsupervised audio pre-training for video-to-speech synthesis [64.86087257004883]
  Video-to-speech synthesis is the task of reconstructing the speech signal from a silent video of a speaker. In this paper we propose to train encoder-decoder models on more than 3,500 hours of audio data at 24kHz. We then use the pre-trained decoders to initialize the audio decoders for the video-to-speech synthesis task.
  arXiv  Detail & Related papers  (2023-06-27T13:31:33Z)
• MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation [70.74377373885645]
  We propose the first joint audio-video generation framework that brings engaging watching and listening experiences simultaneously. MM-Diffusion consists of a sequential multi-modal U-Net for a joint denoising process by design. Experiments show superior results in unconditional audio-video generation, and zero-shot conditional tasks.
  arXiv  Detail & Related papers  (2022-12-19T14:11:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.