Learning to Generate Diverse Dance Motions with Transformer

• URL: http://arxiv.org/abs/2008.08171v1
• Date: Tue, 18 Aug 2020 22:29:40 GMT
• Title: Learning to Generate Diverse Dance Motions with Transformer
• Authors: Jiaman Li, Yihang Yin, Hang Chu, Yi Zhou, Tingwu Wang, Sanja Fidler, Hao Li
• Abstract summary: We introduce a complete system for dance motion synthesis. A massive dance motion data set is created from YouTube videos. A novel two-stream motion transformer generative model can generate motion sequences with high flexibility.
• Score: 67.43270523386185
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: With the ongoing pandemic, virtual concerts and live events using digitized performances of musicians are getting traction on massive multiplayer online worlds. However, well choreographed dance movements are extremely complex to animate and would involve an expensive and tedious production process. In addition to the use of complex motion capture systems, it typically requires a collaborative effort between animators, dancers, and choreographers. We introduce a complete system for dance motion synthesis, which can generate complex and highly diverse dance sequences given an input music sequence. As motion capture data is limited for the range of dance motions and styles, we introduce a massive dance motion data set that is created from YouTube videos. We also present a novel two-stream motion transformer generative model, which can generate motion sequences with high flexibility. We also introduce new evaluation metrics for the quality of synthesized dance motions, and demonstrate that our system can outperform state-of-the-art methods. Our system provides high-quality animations suitable for large crowds for virtual concerts and can also be used as reference for professional animation pipelines. Most importantly, we show that vast online videos can be effective in training dance motion models.

Related papers

• Bidirectional Autoregressive Diffusion Model for Dance Generation [26.449135437337034]
  We propose a Bidirectional Autoregressive Diffusion Model (BADM) for music-to-dance generation. A bidirectional encoder is built to enforce that the generated dance is harmonious in both the forward and backward directions. To make the generated dance motion smoother, a local information decoder is built for local motion enhancement.
  arXiv  Detail & Related papers  (2024-02-06T19:42:18Z)
• TM2D: Bimodality Driven 3D Dance Generation via Music-Text Integration [75.37311932218773]
  We propose a novel task for generating 3D dance movements that simultaneously incorporate both text and music modalities. Our approach can generate realistic and coherent dance movements conditioned on both text and music while maintaining comparable performance with the two single modalities.
  arXiv  Detail & Related papers  (2023-04-05T12:58:33Z)
• BRACE: The Breakdancing Competition Dataset for Dance Motion Synthesis [123.73677487809418]
  We introduce a new dataset aiming to challenge common assumptions in dance motion synthesis. We focus on breakdancing which features acrobatic moves and tangled postures. Our efforts produced the BRACE dataset, which contains over 3 hours and 30 minutes of densely annotated poses.
  arXiv  Detail & Related papers  (2022-07-20T18:03:54Z)
• Quantized GAN for Complex Music Generation from Dance Videos [48.196705493763986]
  We present Dance2Music-GAN (D2M-GAN), a novel adversarial multi-modal framework that generates musical samples conditioned on dance videos. Our proposed framework takes dance video frames and human body motion as input, and learns to generate music samples that plausibly accompany the corresponding input.
  arXiv  Detail & Related papers  (2022-04-01T17:53:39Z)
• Music-to-Dance Generation with Optimal Transport [48.92483627635586]
  We propose a Music-to-Dance with Optimal Transport Network (MDOT-Net) for learning to generate 3D dance choreographs from music. We introduce an optimal transport distance for evaluating the authenticity of the generated dance distribution and a Gromov-Wasserstein distance to measure the correspondence between the dance distribution and the input music.
  arXiv  Detail & Related papers  (2021-12-03T09:37:26Z)
• Transflower: probabilistic autoregressive dance generation with multimodal attention [31.308435764603658]
  We present a novel probabilistic autoregressive architecture that models the distribution over future poses with a normalizing flow conditioned on previous poses as well as music context. Second, we introduce the currently largest 3D dance-motion dataset, obtained with a variety of motion-capture technologies, and including both professional and casual dancers.
  arXiv  Detail & Related papers  (2021-06-25T20:14:28Z)
• DanceFormer: Music Conditioned 3D Dance Generation with Parametric Motion Transformer [23.51701359698245]
  In this paper, we reformulate it by a two-stage process, ie, a key pose generation and then an in-between parametric motion curve prediction. We propose a large-scale music conditioned 3D dance dataset, called PhantomDance, that is accurately labeled by experienced animators. Experiments demonstrate that the proposed method, even trained by existing datasets, can generate fluent, performative, and music-matched 3D dances.
  arXiv  Detail & Related papers  (2021-03-18T12:17:38Z)
• Dance Revolution: Long-Term Dance Generation with Music via Curriculum Learning [55.854205371307884]
  We formalize the music-conditioned dance generation as a sequence-to-sequence learning problem. We propose a novel curriculum learning strategy to alleviate error accumulation of autoregressive models in long motion sequence generation. Our approach significantly outperforms the existing state-of-the-arts on automatic metrics and human evaluation.
  arXiv  Detail & Related papers  (2020-06-11T00:08:25Z)
• Music2Dance: DanceNet for Music-driven Dance Generation [11.73506542921528]
  We propose a novel autoregressive generative model, DanceNet, to take the style, rhythm and melody of music as the control signals. We capture several synchronized music-dance pairs by professional dancers, and build a high-quality music-dance pair dataset.
  arXiv  Detail & Related papers  (2020-02-02T17:18:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.