DreamX-Creator: Democratizing Native Audio-Video Generation at 2K Resolution

• URL: http://arxiv.org/abs/2608.31106v1
• Date: Mon, 31 Aug 2026 17:11:05 GMT
• Title: DreamX-Creator: Democratizing Native Audio-Video Generation at 2K Resolution
• Abstract summary: We present DreamX-Creator 1.0, a compact native joint audio-video generation system centered on a 7B generator.<n>A unified Audio-Video Data System constructs and filters temporally coherent clips, produces structured multimodal annotations, and organizes clips into capability-oriented data pools.<n>For high-resolution output, our Autoregressive 1-Step 2K Refinement pipeline adapts a bidirectional multi-step teacher into an autoregressive multi-step refiner.
• Score: 16.570672947240855
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent video generators often omit audio or synthesize it in a separate stage, limiting reciprocal modeling of visual dynamics and acoustic events. We present DreamX-Creator 1.0, a compact native joint audio-video generation system centered on a 7B generator. Conditioned on a first frame and a text prompt, the generator jointly denoises modality-specialized audio and video streams. The streams are processed independently in the first half of the network and coupled in the latter half through Gated Cross-Modal Attention, whose token- and head-wise output gates modulate each active cross-modal attention-head output. A unified Audio-Video Data System constructs and filters temporally coherent clips, produces structured multimodal annotations, and organizes clips into capability-oriented data pools. Progressive Joint Training comprises two audio-video pre-training stages followed by High-Quality Finetuning. Audio-Video Reinforcement Learning further post-trains the generator with Modality-Aware Multimodal Feedback that routes video-, audio-, and cross-modal feedback to the corresponding streams. For high-resolution output, our Autoregressive 1-Step 2K Refinement pipeline adapts a bidirectional multi-step teacher into an autoregressive multi-step refiner and distills it into a student requiring one denoising evaluation per temporal chunk. Overall, DreamX-Creator 1.0 achieves native, synchronized audio-video generation with performance competitive with state-of-the-art open-source systems. By releasing our compact 7B generator and 2K Refiner, we seek to democratize native audio-video generation and provide an accessible foundation for future research in unified audio-video generative modeling.

Related papers

• AVTok: 1D Unified Tokenization for Holistic Audio-Video Generation [47.70199516002865]
  We present textbfAVTok, a novel unified tokenizer designated for holistic audio-video generation.<n>AVTok features a dual-stream transformer-based architecture with shared encoder-decoder and modal-specific learnable queries.<n>Experiments demonstrate that AVTok excels both in audio-video reconstruction and when integrated into downstream pipelines for audio-to-video, video-to-audio, and class-conditional joint audio-video generation.
  arXiv  Detail & Related papers  (2026-06-29T18:35:17Z)
• AudioX-Turbo: A Unified Framework for Efficient Anything-to-Audio Generation [111.61779149559645]
  We propose AudioX-Turbo, a framework for anything-to-audio generation that integrates varied multimodal conditions.<n>To support the training of AudioX-Turbo, we construct a large-scale, high-quality dataset, IF-caps-Pro, comprising approximately 9.2M samples.<n>We benchmark AudioX-Turbo across a wide range of tasks, finding that our model achieves superior performance, especially on text-to-audio and text-to-music generation.
  arXiv  Detail & Related papers  (2026-06-10T18:06:27Z)
• Diffusion Models for Joint Audio-Video Generation [51.56484100374058]
  I release two high-quality, paired audio-video datasets.<n>I train the MM-Diffusion architecture from scratch on our datasets.<n>I propose a sequential two-step text-to-audio-video generation pipeline.
  arXiv  Detail & Related papers  (2026-03-17T03:31:37Z)
• LTX-2: Efficient Joint Audio-Visual Foundation Model [3.1804093402153506]
  LTX-2 is an open-source model capable of generating temporally synchronized audiovisual content.<n>We employ a multilingual text encoder for broader prompt understanding.<n>LTX-2 produces rich, coherent audio tracks that follow the characters, environment, style, and emotion of each scene.
  arXiv  Detail & Related papers  (2026-01-06T18:24:41Z)
• HunyuanVideo-Foley: Multimodal Diffusion with Representation Alignment for High-Fidelity Foley Audio Generation [14.921126281071544]
  HunyuanVideo-Foley is an end-to-end text-video-to-audio framework.<n>It synthesizes high-fidelity audio precisely aligned with visual dynamics and semantic context.<n>It achieves new state-of-the-art performance across audio fidelity, visual-semantic alignment, temporal alignment and distribution matching.
  arXiv  Detail & Related papers  (2025-08-23T07:30:18Z)
• AV-Link: Temporally-Aligned Diffusion Features for Cross-Modal Audio-Video Generation [49.6922496382879]
  We propose a unified framework for Video-to-Audio (A2V) and Audio-to-Video (A2V) generation.<n>The key to our framework is a Fusion Block that facilitates bidirectional information exchange between video and audio diffusion models.
  arXiv  Detail & Related papers  (2024-12-19T18:57:21Z)
• Frieren: Efficient Video-to-Audio Generation Network with Rectified Flow Matching [51.70360630470263]
  Video-to-audio (V2A) generation aims to synthesize content-matching audio from silent video.<n>We propose Frieren, a V2A model based on rectified flow matching.<n>Experiments indicate that Frieren achieves state-of-the-art performance in both generation quality and temporal alignment.
  arXiv  Detail & Related papers  (2024-06-01T06:40:22Z)
• MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation [70.74377373885645]
  We propose the first joint audio-video generation framework that brings engaging watching and listening experiences simultaneously. MM-Diffusion consists of a sequential multi-modal U-Net for a joint denoising process by design. Experiments show superior results in unconditional audio-video generation, and zero-shot conditional tasks.
  arXiv  Detail & Related papers  (2022-12-19T14:11:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.