OmniDance: Multimodal Driven Dance Video Generation with Large-scale Internet Data

• URL: http://arxiv.org/abs/2606.30019v1
• Date: Mon, 29 Jun 2026 09:23:53 GMT
• Title: OmniDance: Multimodal Driven Dance Video Generation with Large-scale Internet Data
• Abstract summary: CIPE-Dance is the largest dataset for dance video generation to date, comprising 300k high-quality clips over 400 hours.<n>We propose OmniDance, a framework-level recipe for integrating music into a TI2V foundation model without sacrificing original controllability or visual fidelity.
• Score: 20.242950805452892
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Music-driven dance video generation aims to synthesize expressive human motion that is temporally aligned with music while maintaining high visual fidelity. Despite recent progress, existing methods still face two key limitations: the lack of large-scale, high-quality dance video datasets, and the absence of principled frameworks for integrating music as a complementary conditioning signal into Video Generation Foundation Models. To address these limitations, we introduce CIPE-Dance, a large-scale Internet-sourced dance video dataset with choreography-informed text annotations, constructed via a progressive expert pipeline. To the best of our knowledge, CIPE-Dance is the largest dataset for dance video generation to date, comprising 300k high-quality clips over 400 hours and covering diverse dancers, environments, and dance genres. We further propose OmniDance, a framework-level recipe for integrating music into a TI2V foundation model without sacrificing its original controllability or visual fidelity. Motivated by the complementary roles of text as low-frequency semantics and music as high-frequency temporal dynamics, OmniDance co-designs a depth-aware specialization architecture, an anchored easy-to-hard curriculum learning strategy, and a modality-specialized time-dependent CFG strategy, enabling unified TI2V, MI2V, and MTI2V generation. Extensive experiments on CIPE-Dance demonstrate that OmniDance achieves state-of-the-art performance across all three tasks and exhibits robust multimodal integration capability. Project is available at https://github.com/AMAP-ML/OmniDance.

Related papers

• FlowDance: Music-Driven Dance Video Generation with Parallel Pose and RGB Streams [22.399074928933956]
  FlowDance is a music-driven dance video generation framework.<n>It integrates explicit motion modeling with reference-preserving visual synthesis through parallel pose and RGB streams.<n>It achieves strong performance in both dance motion generation and music-driven dance video synthesis.
  arXiv  Detail & Related papers  (2026-08-16T15:48:53Z)
• TokenDance: Token-to-Token Music-to-Dance Generation with Bidirectional Mamba [5.119197329627647]
  Music-to-dance generation has broad applications in virtual reality, dance education, and digital character animation.<n>We present TokenDance, a two-stage music-to-dance generation framework that explicitly addresses this limitation through dual-modality tokenization and efficient token-level generation.<n> TokenDance achieves overall state-of-the-art (SOTA) performance in both generation quality and inference speed, highlighting its effectiveness and practical value for real-world music-to-dance applications.
  arXiv  Detail & Related papers  (2026-03-28T15:38:14Z)
• InfiniteDance: Scalable 3D Dance Generation Towards in-the-wild Generalization [54.47757841373109]
  This work aims to push the frontier of generalizable 3D dance generation by scaling up both data and model design.<n>We develop a fully automated pipeline that reconstructs high-fidelity 3D dance motions from monocular videos.<n>To eliminate the physical artifacts prevalent in existing reconstruction methods, we introduce a Foot Restoration Diffusion Model (FRDM) guided by foot-contact and geometric constraints.<n>To enhance robustness under unfamiliar music conditions, we integrate a retrieval-augmented generation (RAG) module that injects reference dance as a prompt.
  arXiv  Detail & Related papers  (2026-03-10T10:10:34Z)
• MACE-Dance: Motion-Appearance Cascaded Experts for Music-Driven Dance Video Generation [20.517753182293095]
  MACE-Dance is a music-driven dance video generation framework with cascaded Mixture-of-appearance (MoE)<n>The Motion Expert performs music-to-3D motion generation while enforcing kinematic plausibility and artistic expressiveness.<n>The Appearance Expert carries out motion- and reference-conditioned video synthesis, preserving visual identity withtemporal coherence.
  arXiv  Detail & Related papers  (2025-12-20T02:34:34Z)
• Seedance 1.5 pro: A Native Audio-Visual Joint Generation Foundation Model [163.56671779593736]
  Seedance 1.5 pro is a foundational model engineered specifically for native, joint audio-video generation.<n>Seedance 1.5 pro distinguishes itself through precise multilingual and dialect lip-syncing, dynamic cinematic camera control, and enhanced narrative coherence.
  arXiv  Detail & Related papers  (2025-12-15T16:36:52Z)
• OpenDance: Multimodal Controllable 3D Dance Generation Using Large-scale Internet Data [11.915309859714675]
  We build OpenDance5D, an extensive human dance dataset comprising over 101 hours across 14 distinct genres.<n>Each sample has five modalities to facilitate robust cross-modal learning: RGB video, audio, 2D keypoints, 3D motion, and fine-grained textual descriptions from human arts.<n>We propose OpenDanceNet, a unified modeling framework for controllable dance generation conditioned on music and arbitrary combinations of text prompts, keypoints, or character positioning.
  arXiv  Detail & Related papers  (2025-06-09T09:05:13Z)
• GCDance: Genre-Controlled Music-Driven 3D Full Body Dance Generation [30.028340528694432]
  GCDance is a framework for genre-specific 3D full-body dance generation conditioned on music and descriptive text.<n>We develop a text-based control mechanism that maps input prompts, explicit genre labels or free-form descriptive text, into genre-specific control signals.<n>To balance the objectives of extracting text-genre information and maintaining high-quality generation results, we propose a novel multi-task optimization strategy.
  arXiv  Detail & Related papers  (2025-02-25T15:53:18Z)
• Every Image Listens, Every Image Dances: Music-Driven Image Animation [8.085267959520843]
  MuseDance is an end-to-end model that animates reference images using both music and text inputs.<n>Unlike existing approaches, MuseDance eliminates the need for complex motion guidance inputs, such as pose or depth sequences.<n>We present a new multimodal dataset comprising 2,904 dance videos with corresponding background music and text descriptions.
  arXiv  Detail & Related papers  (2025-01-30T23:38:51Z)
• Controllable Dance Generation with Style-Guided Motion Diffusion [49.35282418951445]
  Dance plays an important role as an artistic form and expression in human culture, yet the creation of dance remains a challenging task.<n>Most dance generation methods rely solely on music, seldom taking into consideration intrinsic attributes such as music style or genre.<n>In this work, we introduce Flexible Dance Generation with Style Description Prompts (DGSDP), a diffusion-based framework suitable for diversified tasks of dance generation.
  arXiv  Detail & Related papers  (2024-06-12T04:55:14Z)
• BRACE: The Breakdancing Competition Dataset for Dance Motion Synthesis [123.73677487809418]
  We introduce a new dataset aiming to challenge common assumptions in dance motion synthesis. We focus on breakdancing which features acrobatic moves and tangled postures. Our efforts produced the BRACE dataset, which contains over 3 hours and 30 minutes of densely annotated poses.
  arXiv  Detail & Related papers  (2022-07-20T18:03:54Z)
• Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memory [92.81383016482813]
  We propose a novel music-to-dance framework, Bailando, for driving 3D characters to dance following a piece of music. We introduce an actor-critic Generative Pre-trained Transformer (GPT) that composes units to a fluent dance coherent to the music. Our proposed framework achieves state-of-the-art performance both qualitatively and quantitatively.
  arXiv  Detail & Related papers  (2022-03-24T13:06:43Z)
• Learning to Generate Diverse Dance Motions with Transformer [67.43270523386185]
  We introduce a complete system for dance motion synthesis. A massive dance motion data set is created from YouTube videos. A novel two-stream motion transformer generative model can generate motion sequences with high flexibility.
  arXiv  Detail & Related papers  (2020-08-18T22:29:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.