CineDance: Towards Next-Generation Multi-Shot Long-Form Cinematic Audio-Video Generation

• URL: http://arxiv.org/abs/2606.09639v2
• Date: Thu, 11 Jun 2026 05:58:22 GMT
• Title: CineDance: Towards Next-Generation Multi-Shot Long-Form Cinematic Audio-Video Generation
• Abstract summary: We introduce CineDance-1M, a large-scale, open research Text-to-Audio-Video dataset.<n>Averaging 92.8 seconds and 24.2 continuous shots per video, it provides, structured annotations for both audio and video modalities.<n>For a comprehensive assessment, we propose CineBench, featuring a diverse prompt suite and a six-dimensional, human-aligned metric system.
• Score: 107.75241387090271
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The fidelity and structural diversity of training datasets fundamentally determine the capabilities of video generation models. While commercial systems showremarkableabilitytogeneratecinematicnarratives, the progress of open-source models remains limited by the scarcity of high-quality training data. To bridge this gap, we introduce CineDance-1M, a large-scale, open research Text-to-Audio-Video (T2AV) dataset designed specifically for multi-shot, long-form joint audio-video generation. Averaging 92.8 seconds and 24.2 continuous shots per video, it provides configurable, structured annotations for both audio and video modalities. This exceptional quality is achieved through a rigorous three-stage curation pipeline: i) diverse sourcing and comprehensive cleansing, ii) film-theory-inspired narrative parsing, and iii) hierarchical dual-modal captioning. For a comprehensive assessment, we propose CineBench, featuring a diverse prompt suite and a six-dimensional, human-aligned metric system tailored for complex narrative audio-video evaluation. Furthermore, we adapt LTX-2.3 into CineDance, which demonstrates exceptional single-modality quality alongside precise audio-video alignment and robust subject and environment consistency, effectively validating our curation strategy and the high quality of CineDance-1M. We anticipate that this work will serve as a solid foundation for accelerating future research in multi-shot, long-form joint audio-video generation. Our project page is available at https://aliothchen.github.io/projects/CineDance/.

Related papers

• Diffusion Models for Joint Audio-Video Generation [51.56484100374058]
  I release two high-quality, paired audio-video datasets.<n>I train the MM-Diffusion architecture from scratch on our datasets.<n>I propose a sequential two-step text-to-audio-video generation pipeline.
  arXiv  Detail & Related papers  (2026-03-17T03:31:37Z)
• JavisDiT++: Unified Modeling and Optimization for Joint Audio-Video Generation [112.614973927778]
  Joint audio-video generation (JAVG) produces synchronized and semantically aligned sound and vision from textual descriptions.<n>This paper presents JavisDiT++, a framework for unified modeling and optimization of JAVG.<n>Our model achieves state-of-the-art performance merely with around 1M public training entries.
  arXiv  Detail & Related papers  (2026-02-22T12:44:28Z)
• ALIVE: Animate Your World with Lifelike Audio-Video Generation [50.693986608051716]
  ALIVE is a generation model that adapts a pretrained Text-to-Video (T2V) model to Sora-style audio-video generation and animation.<n>To support the audio-visual synchronization and reference animation, we augment the popular MMDiT architecture with a joint audio-video branch.<n>ALIVE demonstrates outstanding performance, consistently outperforming open-source models and matching or surpassing state-of-the-art commercial solutions.
  arXiv  Detail & Related papers  (2026-02-09T14:06:03Z)
• LTX-2: Efficient Joint Audio-Visual Foundation Model [3.1804093402153506]
  LTX-2 is an open-source model capable of generating temporally synchronized audiovisual content.<n>We employ a multilingual text encoder for broader prompt understanding.<n>LTX-2 produces rich, coherent audio tracks that follow the characters, environment, style, and emotion of each scene.
  arXiv  Detail & Related papers  (2026-01-06T18:24:41Z)
• VABench: A Comprehensive Benchmark for Audio-Video Generation [22.00633729850902]
  VABench is a benchmark framework designed to evaluate the capabilities of synchronous audio-video generation.<n>It covers three task types: text-to-audio-video (T2AV), image-to-audio-video (I2AV), and stereo audio-video generation.<n>VABench covers seven major content categories: animals, human sounds, music, environmental sounds, synchronous physical sounds, complex scenes, and virtual worlds.
  arXiv  Detail & Related papers  (2025-12-10T03:57:29Z)
• TalkCuts: A Large-Scale Dataset for Multi-Shot Human Speech Video Generation [76.48551690189406]
  We present TalkCuts, a large-scale dataset designed to facilitate the study of multi-shot human speech video generation.<n>TalkCuts offers 164k clips totaling over 500 hours of high-quality human speech videos with diverse camera shots, including close-up, half-body, and full-body views.<n>The dataset includes detailed textual descriptions, 2D keypoints and 3D SMPL-X motion annotations, covering over 10k identities, enabling multimodal learning and evaluation.
  arXiv  Detail & Related papers  (2025-10-08T17:16:09Z)
• HunyuanVideo-Foley: Multimodal Diffusion with Representation Alignment for High-Fidelity Foley Audio Generation [14.921126281071544]
  HunyuanVideo-Foley is an end-to-end text-video-to-audio framework.<n>It synthesizes high-fidelity audio precisely aligned with visual dynamics and semantic context.<n>It achieves new state-of-the-art performance across audio fidelity, visual-semantic alignment, temporal alignment and distribution matching.
  arXiv  Detail & Related papers  (2025-08-23T07:30:18Z)
• VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs [55.82090875098132]
  VideoLLaMA 2 is a set of Video Large Language Models (Video-LLMs) designed to enhance spatial-temporal modeling and audio understanding in video and audio-oriented tasks. VideoLLaMA 2 consistently achieves competitive results among open-source models and even gets close to some proprietary models on several benchmarks.
  arXiv  Detail & Related papers  (2024-06-11T17:22:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.