Dual Learning Music Composition and Dance Choreography

• URL: http://arxiv.org/abs/2201.11999v1
• Date: Fri, 28 Jan 2022 09:20:28 GMT
• Title: Dual Learning Music Composition and Dance Choreography
• Authors: Shuang Wu, Zhenguang Li, Shijian Lu, Li Cheng
• Abstract summary: Music and dance have always co-existed as pillars of human activities, contributing immensely to cultural, social, and entertainment functions. Recent research works have studied generative models for dance sequences conditioned on music. We propose a novel extension, where we jointly model both tasks in a dual learning approach.
• Score: 57.55406449959893
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Music and dance have always co-existed as pillars of human activities, contributing immensely to the cultural, social, and entertainment functions in virtually all societies. Notwithstanding the gradual systematization of music and dance into two independent disciplines, their intimate connection is undeniable and one art-form often appears incomplete without the other. Recent research works have studied generative models for dance sequences conditioned on music. The dual task of composing music for given dances, however, has been largely overlooked. In this paper, we propose a novel extension, where we jointly model both tasks in a dual learning approach. To leverage the duality of the two modalities, we introduce an optimal transport objective to align feature embeddings, as well as a cycle consistency loss to foster overall consistency. Experimental results demonstrate that our dual learning framework improves individual task performance, delivering generated music compositions and dance choreographs that are realistic and faithful to the conditioned inputs.

Related papers

• Lodge++: High-quality and Long Dance Generation with Vivid Choreography Patterns [48.54956784928394]
  Lodge++ is a choreography framework to generate high-quality, ultra-long, and vivid dances given the music and desired genre. To handle the challenges in computational efficiency, Lodge++ adopts a two-stage strategy to produce dances from coarse to fine. Lodge++ is validated by extensive experiments, which show that our method can rapidly generate ultra-long dances suitable for various dance genres.
  arXiv  Detail & Related papers  (2024-10-27T09:32:35Z)
• Duolando: Follower GPT with Off-Policy Reinforcement Learning for Dance Accompaniment [87.20240797625648]
  We introduce a novel task within the field of 3D dance generation, termed dance accompaniment. It requires the generation of responsive movements from a dance partner, the "follower", synchronized with the lead dancer's movements and the underlying musical rhythm. We propose a GPT-based model, Duolando, which autoregressively predicts the subsequent tokenized motion conditioned on the coordinated information of the music, the leader's and the follower's movements.
  arXiv  Detail & Related papers  (2024-03-27T17:57:02Z)
• LM2D: Lyrics- and Music-Driven Dance Synthesis [28.884929875333846]
  LM2D is designed to create dance conditioned on both music and lyrics in one diffusion generation step. We introduce the first 3D dance-motion dataset that encompasses both music and lyrics, obtained with pose estimation technologies. The results demonstrate LM2D is able to produce realistic and diverse dance matching both lyrics and music.
  arXiv  Detail & Related papers  (2024-03-14T13:59:04Z)
• Learning Music-Dance Representations through Explicit-Implicit Rhythm Synchronization [22.279424952432677]
  Music-dance representation can be applied to three downstream tasks: (a) dance classification, (b) music-dance retrieval, and (c) music-dance. We derive the dance rhythms based on visual appearance and motion cues inspired by the music rhythm analysis. Then the visual rhythms are temporally aligned with the music counterparts, which are extracted by the amplitude of sound intensity.
  arXiv  Detail & Related papers  (2022-07-07T09:44:44Z)
• Quantized GAN for Complex Music Generation from Dance Videos [48.196705493763986]
  We present Dance2Music-GAN (D2M-GAN), a novel adversarial multi-modal framework that generates musical samples conditioned on dance videos. Our proposed framework takes dance video frames and human body motion as input, and learns to generate music samples that plausibly accompany the corresponding input.
  arXiv  Detail & Related papers  (2022-04-01T17:53:39Z)
• Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memory [92.81383016482813]
  We propose a novel music-to-dance framework, Bailando, for driving 3D characters to dance following a piece of music. We introduce an actor-critic Generative Pre-trained Transformer (GPT) that composes units to a fluent dance coherent to the music. Our proposed framework achieves state-of-the-art performance both qualitatively and quantitatively.
  arXiv  Detail & Related papers  (2022-03-24T13:06:43Z)
• Music-to-Dance Generation with Optimal Transport [48.92483627635586]
  We propose a Music-to-Dance with Optimal Transport Network (MDOT-Net) for learning to generate 3D dance choreographs from music. We introduce an optimal transport distance for evaluating the authenticity of the generated dance distribution and a Gromov-Wasserstein distance to measure the correspondence between the dance distribution and the input music.
  arXiv  Detail & Related papers  (2021-12-03T09:37:26Z)
• ChoreoNet: Towards Music to Dance Synthesis with Choreographic Action Unit [28.877908457607678]
  We design a two-stage music-to-dance synthesis framework ChoreoNet to imitate human choreography procedure. Our framework firstly devises a CAU prediction model to learn the mapping relationship between music and CAU sequences. We then devise a spatial-temporal inpainting model to convert the CAU sequence into continuous dance motions.
  arXiv  Detail & Related papers  (2020-09-16T12:38:19Z)
• Music2Dance: DanceNet for Music-driven Dance Generation [11.73506542921528]
  We propose a novel autoregressive generative model, DanceNet, to take the style, rhythm and melody of music as the control signals. We capture several synchronized music-dance pairs by professional dancers, and build a high-quality music-dance pair dataset.
  arXiv  Detail & Related papers  (2020-02-02T17:18:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.