Automatic Dance Video Segmentation for Understanding Choreography

• URL: http://arxiv.org/abs/2405.19727v1
• Date: Thu, 30 May 2024 06:19:01 GMT
• Title: Automatic Dance Video Segmentation for Understanding Choreography
• Authors: Koki Endo, Shuhei Tsuchida, Tsukasa Fukusato, Takeo Igarashi,
• Abstract summary: We propose a method to automatically segment a dance video into each movement. To build our training dataset, we annotate segmentation points to dance videos in the AIST Dance Video Database. The evaluation study shows that the proposed method can estimate segmentation points with high accuracy.
• Score: 10.053913399613764
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Segmenting dance video into short movements is a popular way to easily understand dance choreography. However, it is currently done manually and requires a significant amount of effort by experts. That is, even if many dance videos are available on social media (e.g., TikTok and YouTube), it remains difficult for people, especially novices, to casually watch short video segments to practice dance choreography. In this paper, we propose a method to automatically segment a dance video into each movement. Given a dance video as input, we first extract visual and audio features: the former is computed from the keypoints of the dancer in the video, and the latter is computed from the Mel spectrogram of the music in the video. Next, these features are passed to a Temporal Convolutional Network (TCN), and segmentation points are estimated by picking peaks of the network output. To build our training dataset, we annotate segmentation points to dance videos in the AIST Dance Video Database, which is a shared database containing original street dance videos with copyright-cleared dance music. The evaluation study shows that the proposed method (i.e., combining the visual and audio features) can estimate segmentation points with high accuracy. In addition, we developed an application to help dancers practice choreography using the proposed method.

Related papers

• Synergy and Synchrony in Couple Dances [62.88254856013913]
  We study what extent social interaction influences one's behavior in the setting of two dancers dancing as a couple. We first consider a baseline in which we predict a dancer's future moves conditioned only on their past motion without regard to their partner. We then investigate the advantage of taking social information into account by conditioning also on the motion of their dancing partner.
  arXiv  Detail & Related papers  (2024-09-06T17:59:01Z)
• Dance Any Beat: Blending Beats with Visuals in Dance Video Generation [12.018432669719742]
  We introduce a novel task: generating dance videos directly from images of individuals guided by music. Our solution, the Dance Any Beat Diffusion model (DabFusion), utilizes a reference image and a music piece to generate dance videos. We evaluate DabFusion's performance using the AIST++ dataset, focusing on video quality, audio-video synchronization, and motion-music alignment.
  arXiv  Detail & Related papers  (2024-05-15T11:33:07Z)
• DanceCamera3D: 3D Camera Movement Synthesis with Music and Dance [50.01162760878841]
  We present DCM, a new multi-modal 3D dataset that combines camera movement with dance motion and music audio. This dataset encompasses 108 dance sequences (3.2 hours) of paired dance-camera-music data from the anime community. We propose DanceCamera3D, a transformer-based diffusion model that incorporates a novel body attention loss and a condition separation strategy.
  arXiv  Detail & Related papers  (2024-03-20T15:24:57Z)
• Music-Driven Group Choreography [10.501572863039852]
  $rm AIOZ-GDANCE$ is a new large-scale dataset for music-driven group dance generation. We show that naively applying single dance generation technique to creating group dance motion may lead to unsatisfactory results. We propose a new method that takes an input music sequence and a set of 3D positions of dancers to efficiently produce multiple group-coherent choreographies.
  arXiv  Detail & Related papers  (2023-03-22T06:26:56Z)
• PirouNet: Creating Intentional Dance with Semi-Supervised Conditional Recurrent Variational Autoencoders [3.867363075280544]
  We propose "PirouNet", a semi-supervised conditional recurrent variational autoencoder with a dance labeling web application. Thanks to the proposed semi-supervised approach, PirouNet only requires a small portion of the dataset to be labeled, typically on the order of 1%. We extensively evaluate PirouNet's dance creations through a series of qualitative and quantitative metrics, validating its applicability as a tool for choreographers.
  arXiv  Detail & Related papers  (2022-07-21T18:04:59Z)
• BRACE: The Breakdancing Competition Dataset for Dance Motion Synthesis [123.73677487809418]
  We introduce a new dataset aiming to challenge common assumptions in dance motion synthesis. We focus on breakdancing which features acrobatic moves and tangled postures. Our efforts produced the BRACE dataset, which contains over 3 hours and 30 minutes of densely annotated poses.
  arXiv  Detail & Related papers  (2022-07-20T18:03:54Z)
• Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memory [92.81383016482813]
  We propose a novel music-to-dance framework, Bailando, for driving 3D characters to dance following a piece of music. We introduce an actor-critic Generative Pre-trained Transformer (GPT) that composes units to a fluent dance coherent to the music. Our proposed framework achieves state-of-the-art performance both qualitatively and quantitatively.
  arXiv  Detail & Related papers  (2022-03-24T13:06:43Z)
• MetaDance: Few-shot Dancing Video Retargeting via Temporal-aware Meta-learning [51.78302763617991]
  Dancing video aims to synthesize a video that transfers the dance movements from a source video to a target person. Previous work need collect a several-minute-long video of a target person with thousands of frames to train a personalized model. Recent work tackled few-shot dancing video, which learns to synthesize videos of unseen persons by leveraging a few frames of them.
  arXiv  Detail & Related papers  (2022-01-13T09:34:20Z)
• Semi-Supervised Learning for In-Game Expert-Level Music-to-Dance Translation [0.0]
  Music-to-dance translation is a powerful feature in recent role-playing games. We re-formulate the translation problem as a piece-wise dance phrase retrieval problem based on the choreography theory. Our method generalizes well over various styles of music and succeeds in expert-level choreography for game players.
  arXiv  Detail & Related papers  (2020-09-27T07:08:04Z)
• DanceIt: Music-inspired Dancing Video Synthesis [38.87762996956861]
  We propose to reproduce such an inherent capability of the human-being within a computer vision system. The proposed system consists of three modules. The generated dancing videos match the content and rhythm of the music.
  arXiv  Detail & Related papers  (2020-09-17T02:29:13Z)
• Learning to Generate Diverse Dance Motions with Transformer [67.43270523386185]
  We introduce a complete system for dance motion synthesis. A massive dance motion data set is created from YouTube videos. A novel two-stream motion transformer generative model can generate motion sequences with high flexibility.
  arXiv  Detail & Related papers  (2020-08-18T22:29:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.