DanceIt: Music-inspired Dancing Video Synthesis
- URL: http://arxiv.org/abs/2009.08027v2
- Date: Sat, 7 Aug 2021 09:14:40 GMT
- Title: DanceIt: Music-inspired Dancing Video Synthesis
- Authors: Xin Guo, Yifan Zhao, Jia Li
- Abstract summary: We propose to reproduce such an inherent capability of the human-being within a computer vision system.
The proposed system consists of three modules.
The generated dancing videos match the content and rhythm of the music.
- Score: 38.87762996956861
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Close your eyes and listen to music, one can easily imagine an actor dancing
rhythmically along with the music. These dance movements are usually made up of
dance movements you have seen before. In this paper, we propose to reproduce
such an inherent capability of the human-being within a computer vision system.
The proposed system consists of three modules. To explore the relationship
between music and dance movements, we propose a cross-modal alignment module
that focuses on dancing video clips, accompanied on pre-designed music, to
learn a system that can judge the consistency between the visual features of
pose sequences and the acoustic features of music. The learned model is then
used in the imagination module to select a pose sequence for the given music.
Such pose sequence selected from the music, however, is usually discontinuous.
To solve this problem, in the spatial-temporal alignment module we develop a
spatial alignment algorithm based on the tendency and periodicity of dance
movements to predict dance movements between discontinuous fragments. In
addition, the selected pose sequence is often misaligned with the music beat.
To solve this problem, we further develop a temporal alignment algorithm to
align the rhythm of music and dance. Finally, the processed pose sequence is
used to synthesize realistic dancing videos in the imagination module. The
generated dancing videos match the content and rhythm of the music.
Experimental results and subjective evaluations show that the proposed approach
can perform the function of generating promising dancing videos by inputting
music.
Related papers
- May the Dance be with You: Dance Generation Framework for Non-Humanoids [6.029154142020362]
We propose a framework for non-humanoid agents to learn how to dance from human videos.
Our framework works in two processes: (1) training a reward model which perceives the relationship between optical flow and music.
Experiment results show that generated dance motion can align with the music beat properly.
arXiv Detail & Related papers (2024-05-30T06:43:55Z) - Dance Any Beat: Blending Beats with Visuals in Dance Video Generation [12.018432669719742]
We introduce a novel task: generating dance videos directly from images of individuals guided by music.
Our solution, the Dance Any Beat Diffusion model (DabFusion), utilizes a reference image and a music piece to generate dance videos.
We evaluate DabFusion's performance using the AIST++ dataset, focusing on video quality, audio-video synchronization, and motion-music alignment.
arXiv Detail & Related papers (2024-05-15T11:33:07Z) - Learning Music-Dance Representations through Explicit-Implicit Rhythm
Synchronization [22.279424952432677]
Music-dance representation can be applied to three downstream tasks: (a) dance classification, (b) music-dance retrieval, and (c) music-dance.
We derive the dance rhythms based on visual appearance and motion cues inspired by the music rhythm analysis. Then the visual rhythms are temporally aligned with the music counterparts, which are extracted by the amplitude of sound intensity.
arXiv Detail & Related papers (2022-07-07T09:44:44Z) - Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic
Memory [92.81383016482813]
We propose a novel music-to-dance framework, Bailando, for driving 3D characters to dance following a piece of music.
We introduce an actor-critic Generative Pre-trained Transformer (GPT) that composes units to a fluent dance coherent to the music.
Our proposed framework achieves state-of-the-art performance both qualitatively and quantitatively.
arXiv Detail & Related papers (2022-03-24T13:06:43Z) - Music-to-Dance Generation with Optimal Transport [48.92483627635586]
We propose a Music-to-Dance with Optimal Transport Network (MDOT-Net) for learning to generate 3D dance choreographs from music.
We introduce an optimal transport distance for evaluating the authenticity of the generated dance distribution and a Gromov-Wasserstein distance to measure the correspondence between the dance distribution and the input music.
arXiv Detail & Related papers (2021-12-03T09:37:26Z) - Learning to Generate Diverse Dance Motions with Transformer [67.43270523386185]
We introduce a complete system for dance motion synthesis.
A massive dance motion data set is created from YouTube videos.
A novel two-stream motion transformer generative model can generate motion sequences with high flexibility.
arXiv Detail & Related papers (2020-08-18T22:29:40Z) - Feel The Music: Automatically Generating A Dance For An Input Song [58.653867648572]
We present a general computational approach that enables a machine to generate a dance for any input music.
We encode intuitive, flexibles for what a 'good' dance is: the structure of the dance should align with the structure of the music.
arXiv Detail & Related papers (2020-06-21T20:29:50Z) - Dance Revolution: Long-Term Dance Generation with Music via Curriculum
Learning [55.854205371307884]
We formalize the music-conditioned dance generation as a sequence-to-sequence learning problem.
We propose a novel curriculum learning strategy to alleviate error accumulation of autoregressive models in long motion sequence generation.
Our approach significantly outperforms the existing state-of-the-arts on automatic metrics and human evaluation.
arXiv Detail & Related papers (2020-06-11T00:08:25Z) - Music2Dance: DanceNet for Music-driven Dance Generation [11.73506542921528]
We propose a novel autoregressive generative model, DanceNet, to take the style, rhythm and melody of music as the control signals.
We capture several synchronized music-dance pairs by professional dancers, and build a high-quality music-dance pair dataset.
arXiv Detail & Related papers (2020-02-02T17:18:31Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.