GCDance: Genre-Controlled 3D Full Body Dance Generation Driven By Music

• URL: http://arxiv.org/abs/2502.18309v1
• Date: Tue, 25 Feb 2025 15:53:18 GMT
• Title: GCDance: Genre-Controlled 3D Full Body Dance Generation Driven By Music
• Authors: Xinran Liu, Xu Dong, Diptesh Kanojia, Wenwu Wang, Zhenhua Feng,
• Abstract summary: GCDance is a classifier-free diffusion framework for generating genre-specific dance motions conditioned on both music and textual prompts.<n>Our approach extracts music features by combining high-level pre-trained music foundation model features with hand-crafted features for multi-granularity feature fusion.
• Score: 22.352036716156967
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Generating high-quality full-body dance sequences from music is a challenging task as it requires strict adherence to genre-specific choreography. Moreover, the generated sequences must be both physically realistic and precisely synchronized with the beats and rhythm of the music. To overcome these challenges, we propose GCDance, a classifier-free diffusion framework for generating genre-specific dance motions conditioned on both music and textual prompts. Specifically, our approach extracts music features by combining high-level pre-trained music foundation model features with hand-crafted features for multi-granularity feature fusion. To achieve genre controllability, we leverage CLIP to efficiently embed genre-based textual prompt representations at each time step within our dance generation pipeline. Our GCDance framework can generate diverse dance styles from the same piece of music while ensuring coherence with the rhythm and melody of the music. Extensive experimental results obtained on the FineDance dataset demonstrate that GCDance significantly outperforms the existing state-of-the-art approaches, which also achieve competitive results on the AIST++ dataset. Our ablation and inference time analysis demonstrate that GCDance provides an effective solution for high-quality music-driven dance generation.

Related papers

• Lodge++: High-quality and Long Dance Generation with Vivid Choreography Patterns [48.54956784928394]
  Lodge++ is a choreography framework to generate high-quality, ultra-long, and vivid dances given the music and desired genre. To handle the challenges in computational efficiency, Lodge++ adopts a two-stage strategy to produce dances from coarse to fine. Lodge++ is validated by extensive experiments, which show that our method can rapidly generate ultra-long dances suitable for various dance genres.
  arXiv  Detail & Related papers  (2024-10-27T09:32:35Z)
• Flexible Music-Conditioned Dance Generation with Style Description Prompts [41.04549275897979]
  We introduce Flexible Dance Generation with Style Description Prompts (DGSDP), a diffusion-based framework suitable for diversified tasks of dance generation. The core component of this framework is Music-Conditioned Style-Aware Diffusion (MCSAD), which comprises a Transformer-based network and a music Style Modulation module. The proposed framework successfully generates realistic dance sequences that are accurately aligned with music for a variety of tasks such as long-term generation, dance in-betweening, dance inpainting, and etc.
  arXiv  Detail & Related papers  (2024-06-12T04:55:14Z)
• DiffDance: Cascaded Human Motion Diffusion Model for Dance Generation [89.50310360658791]
  We present a novel cascaded motion diffusion model, DiffDance, designed for high-resolution, long-form dance generation. This model comprises a music-to-dance diffusion model and a sequence super-resolution diffusion model. We demonstrate that DiffDance is capable of generating realistic dance sequences that align effectively with the input music.
  arXiv  Detail & Related papers  (2023-08-05T16:18:57Z)
• FineDance: A Fine-grained Choreography Dataset for 3D Full Body Dance Generation [33.9261932800456]
  FineDance is the largest music-dance paired dataset with the most dance genres. To address monotonous and unnatural hand movements existing in previous methods, we propose a full-body dance generation network. To further enhance the genre-matching and long-term stability of generated dances, we propose a Genre&Coherent aware Retrieval Module.
  arXiv  Detail & Related papers  (2022-12-07T16:10:08Z)
• Quantized GAN for Complex Music Generation from Dance Videos [48.196705493763986]
  We present Dance2Music-GAN (D2M-GAN), a novel adversarial multi-modal framework that generates musical samples conditioned on dance videos. Our proposed framework takes dance video frames and human body motion as input, and learns to generate music samples that plausibly accompany the corresponding input.
  arXiv  Detail & Related papers  (2022-04-01T17:53:39Z)
• Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memory [92.81383016482813]
  We propose a novel music-to-dance framework, Bailando, for driving 3D characters to dance following a piece of music. We introduce an actor-critic Generative Pre-trained Transformer (GPT) that composes units to a fluent dance coherent to the music. Our proposed framework achieves state-of-the-art performance both qualitatively and quantitatively.
  arXiv  Detail & Related papers  (2022-03-24T13:06:43Z)
• Dual Learning Music Composition and Dance Choreography [57.55406449959893]
  Music and dance have always co-existed as pillars of human activities, contributing immensely to cultural, social, and entertainment functions. Recent research works have studied generative models for dance sequences conditioned on music. We propose a novel extension, where we jointly model both tasks in a dual learning approach.
  arXiv  Detail & Related papers  (2022-01-28T09:20:28Z)
• Music-to-Dance Generation with Optimal Transport [48.92483627635586]
  We propose a Music-to-Dance with Optimal Transport Network (MDOT-Net) for learning to generate 3D dance choreographs from music. We introduce an optimal transport distance for evaluating the authenticity of the generated dance distribution and a Gromov-Wasserstein distance to measure the correspondence between the dance distribution and the input music.
  arXiv  Detail & Related papers  (2021-12-03T09:37:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.