PF-D2M: A Pose-free Diffusion Model for Universal Dance-to-Music Generation

• URL: http://arxiv.org/abs/2601.15872v1
• Date: Thu, 22 Jan 2026 11:21:54 GMT
• Title: PF-D2M: A Pose-free Diffusion Model for Universal Dance-to-Music Generation
• Abstract summary: PF-D2M is a universal diffusion-based dance-to-music generation model that incorporates visual features extracted from dance videos.<n> PF-D2M achieves state-of-the-art performance in dance-music alignment and music quality.
• Score: 6.621690152957197
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Dance-to-music generation aims to generate music that is aligned with dance movements. Existing approaches typically rely on body motion features extracted from a single human dancer and limited dance-to-music datasets, which restrict their performance and applicability to real-world scenarios involving multiple dancers and non-human dancers. In this paper, we propose PF-D2M, a universal diffusion-based dance-to-music generation model that incorporates visual features extracted from dance videos. PF-D2M is trained with a progressive training strategy that effectively addresses data scarcity and generalization challenges. Both objective and subjective evaluations show that PF-D2M achieves state-of-the-art performance in dance-music alignment and music quality.

Related papers

• DanceDuo: Bridging Human Movement and AI Choreography [4.334626115383444]
  This paper introduces a novel platform, namely DanceDuo, leveraging diffusion models to generate AI-choreographed dance sequences synchronized with a variety of music genres.<n>The system allows users to interact with AI by selecting music tracks, humanoid models, and importing personal dance videos for comparison, fostering a rich and engaging user experience.
  arXiv  Detail & Related papers  (2026-06-25T01:26:36Z)
• TokenDance: Token-to-Token Music-to-Dance Generation with Bidirectional Mamba [5.119197329627647]
  Music-to-dance generation has broad applications in virtual reality, dance education, and digital character animation.<n>We present TokenDance, a two-stage music-to-dance generation framework that explicitly addresses this limitation through dual-modality tokenization and efficient token-level generation.<n> TokenDance achieves overall state-of-the-art (SOTA) performance in both generation quality and inference speed, highlighting its effectiveness and practical value for real-world music-to-dance applications.
  arXiv  Detail & Related papers  (2026-03-28T15:38:14Z)
• InfiniteDance: Scalable 3D Dance Generation Towards in-the-wild Generalization [54.47757841373109]
  This work aims to push the frontier of generalizable 3D dance generation by scaling up both data and model design.<n>We develop a fully automated pipeline that reconstructs high-fidelity 3D dance motions from monocular videos.<n>To eliminate the physical artifacts prevalent in existing reconstruction methods, we introduce a Foot Restoration Diffusion Model (FRDM) guided by foot-contact and geometric constraints.<n>To enhance robustness under unfamiliar music conditions, we integrate a retrieval-augmented generation (RAG) module that injects reference dance as a prompt.
  arXiv  Detail & Related papers  (2026-03-10T10:10:34Z)
• Not Like Transformers: Drop the Beat Representation for Dance Generation with Mamba-Based Diffusion Model [8.542693293312935]
  We propose emphMambaDance, a new dance generation approach that leverages a Mamba-based diffusion model.<n>Mamba, well-suited to handling long and autoregressive sequences, is integrated into our two-stage diffusion architecture, substituting off-the-shelf Transformer.<n>We show that our proposed method effectively generates plausible dance movements while reflecting essential characteristics, consistently from short to long dances, compared to the previous methods.
  arXiv  Detail & Related papers  (2026-03-09T06:59:03Z)
• DanceChat: Large Language Model-Guided Music-to-Dance Generation [8.455652926559427]
  Music-to-dance generation aims to synthesize human dance motion conditioned on musical input.<n>We introduce DanceChat, a Large Language Model (LLM)-guided music-to-dance generation approach.
  arXiv  Detail & Related papers  (2025-06-12T11:03:47Z)
• PAMD: Plausibility-Aware Motion Diffusion Model for Long Dance Generation [51.2555550979386]
  Plausibility-Aware Motion Diffusion (PAMD) is a framework for generating dances that are both musically aligned and physically realistic.<n>To provide more effective guidance during generation, we incorporate Prior Motion Guidance (PMG)<n>Experiments show that PAMD significantly improves musical alignment and enhances the physical plausibility of generated motions.
  arXiv  Detail & Related papers  (2025-05-26T14:44:09Z)
• Controllable Dance Generation with Style-Guided Motion Diffusion [49.35282418951445]
  Dance plays an important role as an artistic form and expression in human culture, yet the creation of dance remains a challenging task.<n>Most dance generation methods rely solely on music, seldom taking into consideration intrinsic attributes such as music style or genre.<n>In this work, we introduce Flexible Dance Generation with Style Description Prompts (DGSDP), a diffusion-based framework suitable for diversified tasks of dance generation.
  arXiv  Detail & Related papers  (2024-06-12T04:55:14Z)
• LM2D: Lyrics- and Music-Driven Dance Synthesis [28.884929875333846]
  LM2D is designed to create dance conditioned on both music and lyrics in one diffusion generation step. We introduce the first 3D dance-motion dataset that encompasses both music and lyrics, obtained with pose estimation technologies. The results demonstrate LM2D is able to produce realistic and diverse dance matching both lyrics and music.
  arXiv  Detail & Related papers  (2024-03-14T13:59:04Z)
• Bidirectional Autoregressive Diffusion Model for Dance Generation [26.449135437337034]
  We propose a Bidirectional Autoregressive Diffusion Model (BADM) for music-to-dance generation. A bidirectional encoder is built to enforce that the generated dance is harmonious in both the forward and backward directions. To make the generated dance motion smoother, a local information decoder is built for local motion enhancement.
  arXiv  Detail & Related papers  (2024-02-06T19:42:18Z)
• DiffDance: Cascaded Human Motion Diffusion Model for Dance Generation [89.50310360658791]
  We present a novel cascaded motion diffusion model, DiffDance, designed for high-resolution, long-form dance generation. This model comprises a music-to-dance diffusion model and a sequence super-resolution diffusion model. We demonstrate that DiffDance is capable of generating realistic dance sequences that align effectively with the input music.
  arXiv  Detail & Related papers  (2023-08-05T16:18:57Z)
• TM2D: Bimodality Driven 3D Dance Generation via Music-Text Integration [75.37311932218773]
  We propose a novel task for generating 3D dance movements that simultaneously incorporate both text and music modalities. Our approach can generate realistic and coherent dance movements conditioned on both text and music while maintaining comparable performance with the two single modalities.
  arXiv  Detail & Related papers  (2023-04-05T12:58:33Z)
• Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memory [92.81383016482813]
  We propose a novel music-to-dance framework, Bailando, for driving 3D characters to dance following a piece of music. We introduce an actor-critic Generative Pre-trained Transformer (GPT) that composes units to a fluent dance coherent to the music. Our proposed framework achieves state-of-the-art performance both qualitatively and quantitatively.
  arXiv  Detail & Related papers  (2022-03-24T13:06:43Z)
• Music-to-Dance Generation with Optimal Transport [48.92483627635586]
  We propose a Music-to-Dance with Optimal Transport Network (MDOT-Net) for learning to generate 3D dance choreographs from music. We introduce an optimal transport distance for evaluating the authenticity of the generated dance distribution and a Gromov-Wasserstein distance to measure the correspondence between the dance distribution and the input music.
  arXiv  Detail & Related papers  (2021-12-03T09:37:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.