DanceCamAnimator: Keyframe-Based Controllable 3D Dance Camera Synthesis

• URL: http://arxiv.org/abs/2409.14925v1
• Date: Mon, 23 Sep 2024 11:20:44 GMT
• Title: DanceCamAnimator: Keyframe-Based Controllable 3D Dance Camera Synthesis
• Authors: Zixuan Wang, Jiayi Li, Xiaoyu Qin, Shikun Sun, Songtao Zhou, Jia Jia, Jiebo Luo,
• Abstract summary: Dance camera movements involve both continuous sequences of variable lengths and sudden changes to simulate the switching of multiple cameras. We propose to integrate cinematography knowledge by formulating this task as a three-stage process: animator detection, synthesis, and tween function prediction. Following this formulation, we design a novel end-to-end dance camera framework textbfDanceCamAnimator, which imitates human animation procedures and shows powerful-based controllability with variable lengths.
• Score: 49.614150163184064
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Synthesizing camera movements from music and dance is highly challenging due to the contradicting requirements and complexities of dance cinematography. Unlike human movements, which are always continuous, dance camera movements involve both continuous sequences of variable lengths and sudden drastic changes to simulate the switching of multiple cameras. However, in previous works, every camera frame is equally treated and this causes jittering and unavoidable smoothing in post-processing. To solve these problems, we propose to integrate animator dance cinematography knowledge by formulating this task as a three-stage process: keyframe detection, keyframe synthesis, and tween function prediction. Following this formulation, we design a novel end-to-end dance camera synthesis framework \textbf{DanceCamAnimator}, which imitates human animation procedures and shows powerful keyframe-based controllability with variable lengths. Extensive experiments on the DCM dataset demonstrate that our method surpasses previous baselines quantitatively and qualitatively. Code will be available at \url{https://github.com/Carmenw1203/DanceCamAnimator-Official}.

Related papers

• Cavia: Camera-controllable Multi-view Video Diffusion with View-Integrated Attention [62.2447324481159]
  Cavia is a novel framework for camera-controllable, multi-view video generation. Our framework extends the spatial and temporal attention modules, improving both viewpoint and temporal consistency. Cavia is the first of its kind that allows the user to specify distinct camera motion while obtaining object motion.
  arXiv  Detail & Related papers  (2024-10-14T17:46:32Z)
• Text-guided 3D Human Motion Generation with Keyframe-based Parallel Skip Transformer [62.29951737214263]
  Existing algorithms directly generate the full sequence which is expensive and prone to errors. We propose KeyMotion, that generates plausible human motion sequences corresponding to input text. We use a Variationalcoder (VAE) with Kullback-Leibler regularization to project the Autoencoder into a latent space. For the reverse diffusion, we propose a novel Parallel Skip Transformer that performs cross-modal attention between the design latents and text condition.
  arXiv  Detail & Related papers  (2024-05-24T11:12:37Z)
• DanceCamera3D: 3D Camera Movement Synthesis with Music and Dance [50.01162760878841]
  We present DCM, a new multi-modal 3D dataset that combines camera movement with dance motion and music audio. This dataset encompasses 108 dance sequences (3.2 hours) of paired dance-camera-music data from the anime community. We propose DanceCamera3D, a transformer-based diffusion model that incorporates a novel body attention loss and a condition separation strategy.
  arXiv  Detail & Related papers  (2024-03-20T15:24:57Z)
• BRACE: The Breakdancing Competition Dataset for Dance Motion Synthesis [123.73677487809418]
  We introduce a new dataset aiming to challenge common assumptions in dance motion synthesis. We focus on breakdancing which features acrobatic moves and tangled postures. Our efforts produced the BRACE dataset, which contains over 3 hours and 30 minutes of densely annotated poses.
  arXiv  Detail & Related papers  (2022-07-20T18:03:54Z)
• Music-driven Dance Regeneration with Controllable Key Pose Constraints [17.05495504855978]
  We propose a novel framework for music-driven dance motion synthesis with controllable key pose constraint. Our model involves two single-modal transformer encoders for music and motion representations and a cross-modal transformer decoder for dance motions generation.
  arXiv  Detail & Related papers  (2022-07-08T04:26:45Z)
• DanceFormer: Music Conditioned 3D Dance Generation with Parametric Motion Transformer [23.51701359698245]
  In this paper, we reformulate it by a two-stage process, ie, a key pose generation and then an in-between parametric motion curve prediction. We propose a large-scale music conditioned 3D dance dataset, called PhantomDance, that is accurately labeled by experienced animators. Experiments demonstrate that the proposed method, even trained by existing datasets, can generate fluent, performative, and music-matched 3D dances.
  arXiv  Detail & Related papers  (2021-03-18T12:17:38Z)
• Learn to Dance with AIST++: Music Conditioned 3D Dance Generation [28.623222697548456]
  We present a transformer-based learning framework for 3D dance generation conditioned on music. We also propose a new dataset of paired 3D motion and music called AIST++, which we reconstruct from the AIST multi-view dance videos.
  arXiv  Detail & Related papers  (2021-01-21T18:59:22Z)
• Learning to Generate Diverse Dance Motions with Transformer [67.43270523386185]
  We introduce a complete system for dance motion synthesis. A massive dance motion data set is created from YouTube videos. A novel two-stream motion transformer generative model can generate motion sequences with high flexibility.
  arXiv  Detail & Related papers  (2020-08-18T22:29:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.