Reframing Music-Driven 2D Dance Pose Generation as Multi-Channel Image Generation

• URL: http://arxiv.org/abs/2512.11720v1
• Date: Fri, 12 Dec 2025 16:57:46 GMT
• Title: Reframing Music-Driven 2D Dance Pose Generation as Multi-Channel Image Generation
• Abstract summary: Recent pose-to-video models can translate 2D pose sequences into photorealistic, identity-preserving dance videos.<n>Key challenge is to generate temporally coherent, rhythm-aligned 2D poses from music.<n>We address this by reframing music-to-dance generation as a music-token multi-channel image synthesis problem.
• Score: 11.649112409950304
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Recent pose-to-video models can translate 2D pose sequences into photorealistic, identity-preserving dance videos, so the key challenge is to generate temporally coherent, rhythm-aligned 2D poses from music, especially under complex, high-variance in-the-wild distributions. We address this by reframing music-to-dance generation as a music-token-conditioned multi-channel image synthesis problem: 2D pose sequences are encoded as one-hot images, compressed by a pretrained image VAE, and modeled with a DiT-style backbone, allowing us to inherit architectural and training advances from modern text-to-image models and better capture high-variance 2D pose distributions. On top of this formulation, we introduce (i) a time-shared temporal indexing scheme that explicitly synchronizes music tokens and pose latents over time and (ii) a reference-pose conditioning strategy that preserves subject-specific body proportions and on-screen scale while enabling long-horizon segment-and-stitch generation. Experiments on a large in-the-wild 2D dance corpus and the calibrated AIST++2D benchmark show consistent improvements over representative music-to-dance methods in pose- and video-space metrics and human preference, and ablations validate the contributions of the representation, temporal indexing, and reference conditioning. See supplementary videos at https://hot-dance.github.io

Related papers

• FlowDance: Music-Driven Dance Video Generation with Parallel Pose and RGB Streams [22.399074928933956]
  FlowDance is a music-driven dance video generation framework.<n>It integrates explicit motion modeling with reference-preserving visual synthesis through parallel pose and RGB streams.<n>It achieves strong performance in both dance motion generation and music-driven dance video synthesis.
  arXiv  Detail & Related papers  (2026-08-16T15:48:53Z)
• FlowLong: Inference-time Long Video Generation via Manifold-constrained Tweedie Matching [68.01498128172214]
  We propose a novel but simple inference-time approach for long video generation that is architecture-agnostic and requires no additional training.<n>Our method generates long videos via overlapping sliding windows, where predicted clean samples from adjacent windows are blended via emphTweedie matching to enforce both textbfmanifold constraint and temporal consistency across overlap regions.<n>Our method generates videos several times longer than the native window length while outperforming both training-free and autoregressive baselines in temporal consistency and visual quality, and further extends to audio-video joint generation and text-to-3DGS without any
  arXiv  Detail & Related papers  (2026-05-20T08:55:37Z)
• InfiniteDance: Scalable 3D Dance Generation Towards in-the-wild Generalization [54.47757841373109]
  This work aims to push the frontier of generalizable 3D dance generation by scaling up both data and model design.<n>We develop a fully automated pipeline that reconstructs high-fidelity 3D dance motions from monocular videos.<n>To eliminate the physical artifacts prevalent in existing reconstruction methods, we introduce a Foot Restoration Diffusion Model (FRDM) guided by foot-contact and geometric constraints.<n>To enhance robustness under unfamiliar music conditions, we integrate a retrieval-augmented generation (RAG) module that injects reference dance as a prompt.
  arXiv  Detail & Related papers  (2026-03-10T10:10:34Z)
• Diff-V2M: A Hierarchical Conditional Diffusion Model with Explicit Rhythmic Modeling for Video-to-Music Generation [26.273309051211204]
  Video-to-music (V2M) generation aims to create music that aligns with visual content.<n>We propose Diff-V2M, a general V2M framework based on a hierarchical conditional diffusion model.<n>For rhythm modeling, we begin by evaluating several rhythmic representations, including low-resolution mel-spectrograms, tempograms, and onset detection functions (ODF)
  arXiv  Detail & Related papers  (2025-11-12T08:02:06Z)
• 4D Driving Scene Generation With Stereo Forcing [62.47705572424127]
  Current generative models struggle to synthesize dynamic 4D driving scenes that simultaneously support temporal extrapolation and spatial novel view synthesis (NVS) without per-scene optimization.<n>We present PhiGenesis, a unified framework for 4D scene generation that extends video generation techniques with geometric and temporal consistency.
  arXiv  Detail & Related papers  (2025-09-24T15:37:17Z)
• DuetGen: Music Driven Two-Person Dance Generation via Hierarchical Masked Modeling [70.79846001735547]
  We present DuetGen, a framework for generating interactive two-person dances from music.<n>Inspired by the recent advances in motion synthesis, we propose a two-stage solution.<n>We represent both dancers' motions as a unified whole to learn the necessary motion tokens.
  arXiv  Detail & Related papers  (2025-06-23T14:22:50Z)
• DynamiCtrl: Rethinking the Basic Structure and the Role of Text for High-quality Human Image Animation [63.781450025764904]
  We propose DynamiCtrl, a novel framework for human animation in video DiT architecture.<n>We use a shared VAE encoder for human images and driving poses, unifying them into a common latent space.<n>We also introduce the "Joint-text" paradigm, which preserves the role of text embeddings to provide global semantic context.
  arXiv  Detail & Related papers  (2025-03-27T08:07:45Z)
• X-Dancer: Expressive Music to Human Dance Video Generation [26.544761204917336]
  X-Dancer is a novel zero-shot music-driven image animation pipeline.<n>It creates diverse and long-range lifelike human dance videos from a single static image.
  arXiv  Detail & Related papers  (2025-02-24T18:47:54Z)
• DreamDance: Animating Human Images by Enriching 3D Geometry Cues from 2D Poses [57.17501809717155]
  We present DreamDance, a novel method for animating human images using only skeleton pose sequences as conditional inputs.<n>Our key insight is that human images naturally exhibit multiple levels of correlation.<n>We construct the TikTok-Dance5K dataset, comprising 5K high-quality dance videos with detailed frame annotations.
  arXiv  Detail & Related papers  (2024-11-30T08:42:13Z)
• Dance Any Beat: Blending Beats with Visuals in Dance Video Generation [12.018432669719742]
  We introduce a novel task: generating dance videos directly from images of individuals guided by music.<n>Our solution, the Dance Any Beat Diffusion model (DabFusion), utilizes a reference image and a music piece to generate dance videos.<n>We evaluate DabFusion's performance using the AIST++ dataset, focusing on video quality, audio-video synchronization, and motion-music alignment.
  arXiv  Detail & Related papers  (2024-05-15T11:33:07Z)
• Exploring Pre-trained Text-to-Video Diffusion Models for Referring Video Object Segmentation [72.90144343056227]
  We explore the visual representations produced from a pre-trained text-to-video (T2V) diffusion model for video understanding tasks. We introduce a novel framework, termed "VD-IT", tailored with dedicatedly designed components built upon a fixed T2V model. Our VD-IT achieves highly competitive results, surpassing many existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-03-18T17:59:58Z)
• Do You Guys Want to Dance: Zero-Shot Compositional Human Dance Generation with Multiple Persons [73.21855272778616]
  We introduce a new task, dataset, and evaluation protocol of compositional human dance generation (cHDG) We propose a novel zero-shot framework, dubbed MultiDance-Zero, that can synthesize videos consistent with arbitrary multiple persons and background while precisely following the driving poses.
  arXiv  Detail & Related papers  (2024-01-24T10:44:16Z)
• MAGMA: Music Aligned Generative Motion Autodecoder [15.825872274297735]
  We introduce a 2-step approach for generating dance using a Vector Quantized-Variational Autoencoder (VQ-VAE) We also evaluate the importance of music representations by comparing naive music feature extraction using Librosa to deep audio representations generated by state-of-the-art audio compression algorithms. Our proposed approach achieve state-of-the-art results in music-to-motion generation benchmarks and enables the real-time generation of considerably longer motion sequences.
  arXiv  Detail & Related papers  (2023-09-03T15:21:47Z)
• DanceFormer: Music Conditioned 3D Dance Generation with Parametric Motion Transformer [23.51701359698245]
  In this paper, we reformulate it by a two-stage process, ie, a key pose generation and then an in-between parametric motion curve prediction. We propose a large-scale music conditioned 3D dance dataset, called PhantomDance, that is accurately labeled by experienced animators. Experiments demonstrate that the proposed method, even trained by existing datasets, can generate fluent, performative, and music-matched 3D dances.
  arXiv  Detail & Related papers  (2021-03-18T12:17:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.