Music- and Lyrics-driven Dance Synthesis

• URL: http://arxiv.org/abs/2310.00455v1
• Date: Sat, 30 Sep 2023 18:27:14 GMT
• Title: Music- and Lyrics-driven Dance Synthesis
• Authors: Wenjie Yin, Qingyuan Yao, Yi Yu, Hang Yin, Danica Kragic, M{\aa}rten Bj\"orkman
• Abstract summary: We introduce JustLMD, a new dataset of 3D dance motion with music and lyrics. To the best of our knowledge, this is the first dataset with triplet information including dance motion, music, and lyrics. The proposed JustLMD dataset encompasses 4.6 hours of 3D dance motion in 1867 sequences, accompanied by musical tracks and their corresponding English lyrics.
• Score: 25.474741654098114
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Lyrics often convey information about the songs that are beyond the auditory dimension, enriching the semantic meaning of movements and musical themes. Such insights are important in the dance choreography domain. However, most existing dance synthesis methods mainly focus on music-to-dance generation, without considering the semantic information. To complement it, we introduce JustLMD, a new multimodal dataset of 3D dance motion with music and lyrics. To the best of our knowledge, this is the first dataset with triplet information including dance motion, music, and lyrics. Additionally, we showcase a cross-modal diffusion-based network designed to generate 3D dance motion conditioned on music and lyrics. The proposed JustLMD dataset encompasses 4.6 hours of 3D dance motion in 1867 sequences, accompanied by musical tracks and their corresponding English lyrics.

Related papers

• DanceCamera3D: 3D Camera Movement Synthesis with Music and Dance [50.01162760878841]
  We present DCM, a new multi-modal 3D dataset that combines camera movement with dance motion and music audio. This dataset encompasses 108 dance sequences (3.2 hours) of paired dance-camera-music data from the anime community. We propose DanceCamera3D, a transformer-based diffusion model that incorporates a novel body attention loss and a condition separation strategy.
  arXiv  Detail & Related papers  (2024-03-20T15:24:57Z)
• LM2D: Lyrics- and Music-Driven Dance Synthesis [28.884929875333846]
  LM2D is designed to create dance conditioned on both music and lyrics in one diffusion generation step. We introduce the first 3D dance-motion dataset that encompasses both music and lyrics, obtained with pose estimation technologies. The results demonstrate LM2D is able to produce realistic and diverse dance matching both lyrics and music.
  arXiv  Detail & Related papers  (2024-03-14T13:59:04Z)
• TM2D: Bimodality Driven 3D Dance Generation via Music-Text Integration [75.37311932218773]
  We propose a novel task for generating 3D dance movements that simultaneously incorporate both text and music modalities. Our approach can generate realistic and coherent dance movements conditioned on both text and music while maintaining comparable performance with the two single modalities.
  arXiv  Detail & Related papers  (2023-04-05T12:58:33Z)
• Music-Driven Group Choreography [10.501572863039852]
  $rm AIOZ-GDANCE$ is a new large-scale dataset for music-driven group dance generation. We show that naively applying single dance generation technique to creating group dance motion may lead to unsatisfactory results. We propose a new method that takes an input music sequence and a set of 3D positions of dancers to efficiently produce multiple group-coherent choreographies.
  arXiv  Detail & Related papers  (2023-03-22T06:26:56Z)
• BRACE: The Breakdancing Competition Dataset for Dance Motion Synthesis [123.73677487809418]
  We introduce a new dataset aiming to challenge common assumptions in dance motion synthesis. We focus on breakdancing which features acrobatic moves and tangled postures. Our efforts produced the BRACE dataset, which contains over 3 hours and 30 minutes of densely annotated poses.
  arXiv  Detail & Related papers  (2022-07-20T18:03:54Z)
• Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memory [92.81383016482813]
  We propose a novel music-to-dance framework, Bailando, for driving 3D characters to dance following a piece of music. We introduce an actor-critic Generative Pre-trained Transformer (GPT) that composes units to a fluent dance coherent to the music. Our proposed framework achieves state-of-the-art performance both qualitatively and quantitatively.
  arXiv  Detail & Related papers  (2022-03-24T13:06:43Z)
• Music-to-Dance Generation with Optimal Transport [48.92483627635586]
  We propose a Music-to-Dance with Optimal Transport Network (MDOT-Net) for learning to generate 3D dance choreographs from music. We introduce an optimal transport distance for evaluating the authenticity of the generated dance distribution and a Gromov-Wasserstein distance to measure the correspondence between the dance distribution and the input music.
  arXiv  Detail & Related papers  (2021-12-03T09:37:26Z)
• Learn to Dance with AIST++: Music Conditioned 3D Dance Generation [28.623222697548456]
  We present a transformer-based learning framework for 3D dance generation conditioned on music. We also propose a new dataset of paired 3D motion and music called AIST++, which we reconstruct from the AIST multi-view dance videos.
  arXiv  Detail & Related papers  (2021-01-21T18:59:22Z)
• Learning to Generate Diverse Dance Motions with Transformer [67.43270523386185]
  We introduce a complete system for dance motion synthesis. A massive dance motion data set is created from YouTube videos. A novel two-stream motion transformer generative model can generate motion sequences with high flexibility.
  arXiv  Detail & Related papers  (2020-08-18T22:29:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.