SingDance: Compositional Zero-Shot Singing-and-Dancing Video Generation with Role-Aware Audio Conditioning

• URL: http://arxiv.org/abs/2608.16220v1
• Date: Mon, 17 Aug 2026 07:53:43 GMT
• Title: SingDance: Compositional Zero-Shot Singing-and-Dancing Video Generation with Role-Aware Audio Conditioning
• Abstract summary: We introduce SingDance, a unified video diffusion framework that formulates controllable vocal articulation as a semantic role.<n>We show strong motion--beat alignment and visual fidelity, reliable paired switching of vocal articulation while preserving music-aligned body motion, and highly competitive lip synchronization with substantially fewer generation-time parameters than the strongest speech-driven baseline evaluated.
• Score: 36.11598763279861
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Generating personalized dance videos from a reference image, text prompt, and audio track requires music-conditioned body motion. Singing-and-dancing adds a second requirement: the visible subject must also articulate the vocals. Existing music-conditioned methods focus primarily on choreography, while speech-driven models generally assume that the visible subject produces the input voice, leaving this combined setting largely underexplored. We introduce SingDance, a unified video diffusion framework that formulates controllable vocal articulation as a semantic role: the visible subject is either the source, who produces the vocal signal, or the listener, who receives it from an off-screen performer. Hard-compact routing selects task-relevant speech, music, and role conditions, which are composed through frame-wise joint audio injection; source and listener retain the same speech pathway. Training uses asymmetric supervision: on-screen speaking and curated off-screen conversational-response videos establish role control, while instrumental and song-based dancing-only videos establish music-conditioned body motion. The target Song/Source configuration is never observed during training. At inference, assigning the source role to a song composes separately learned articulation and song-conditioned dance capabilities, enabling compositional zero-shot singing-and-dancing. Experiments demonstrate strong motion--beat alignment and visual fidelity, reliable paired switching of vocal articulation while preserving music-aligned body motion, and highly competitive lip synchronization with substantially fewer generation-time parameters than the strongest speech-driven baseline evaluated.

Related papers

• InterSing: Explicit Interaction Dynamics for 3D Duet Singing Animation and Beyond [56.764269093059504]
  InterSing is a framework for generating realistic 3D head animations for duet singing performances.<n>InterSing generates realistic and expressive singing head animations with stronger coordination and musical alignment than existing methods.
  arXiv  Detail & Related papers  (2026-09-04T09:02:52Z)
• ReFree: Towards Realistic Co-Speech Video Generation via Reward-Free RL and Multilevel Speech Guidance [50.482989497576476]
  Speech-driven talking character animation seeks to generate life-like portrait videos that convey natural conversation behavior.<n>Existing approaches typically trade off precise phoneme-to-lip synchronization against dynamic facial expressions and head motion.<n>We propose ReFree-S2V, a flow-matching speech-to-portrait animation framework that builds upon a pretrained video generation model.
  arXiv  Detail & Related papers  (2026-06-11T13:00:58Z)
• TeMuDance: Contrastive Alignment-Based Textual Control for Music-Driven Dance Generation [32.964897427586166]
  TeMuDance is a framework that enables text-based control for music-conditioned dance generation.<n>It uses motion as a shared semantic anchor to align disjoint music-dance and text-motion datasets.<n>It achieves competitive dance quality while substantially improving text-conditioned control over existing methods.
  arXiv  Detail & Related papers  (2026-04-18T14:24:57Z)
• Think2Sing: Orchestrating Structured Motion Subtitles for Singing-Driven 3D Head Animation [69.50178144839275]
  Singing involves richer emotional nuance, dynamic prosody, and lyric-based semantics.<n>Existing speech-driven approaches often produce oversimplified, emotionally flat, and semantically inconsistent results.<n>Think2Sing generates semantically coherent and temporally consistent 3D head animations conditioned on both lyrics and acoustics.
  arXiv  Detail & Related papers  (2025-09-02T12:59:27Z)
• ChoreoMuse: Robust Music-to-Dance Video Generation with Style Transfer and Beat-Adherent Motion [10.21851621470535]
  We introduce ChoreoMuse, a diffusion-based framework that uses SMPL format parameters and their variation version as intermediaries between music and video generation.<n>ChoreoMuse supports style-controllable, high-fidelity dance video generation across diverse musical genres and individual dancer characteristics.<n>Our method employs a novel music encoder MotionTune to capture motion cues from audio, ensuring that the generated choreography closely follows the beat and expressive qualities of the input music.
  arXiv  Detail & Related papers  (2025-07-26T07:17:50Z)
• SayAnything: Audio-Driven Lip Synchronization with Conditional Video Diffusion [78.77211425667542]
  SayAnything is a conditional video diffusion framework that directly synthesizes lip movements from audio input.<n>Our novel design effectively balances different condition signals in the latent space, enabling precise control over appearance, motion, and region-specific generation.
  arXiv  Detail & Related papers  (2025-02-17T07:29:36Z)
• StyleTalker: One-shot Style-based Audio-driven Talking Head Video Generation [47.06075725469252]
  StyleTalker is an audio-driven talking head generation model. It can synthesize a video of a talking person from a single reference image. Our model is able to synthesize talking head videos with impressive perceptual quality.
  arXiv  Detail & Related papers  (2022-08-23T12:49:01Z)
• Learning Music-Dance Representations through Explicit-Implicit Rhythm Synchronization [22.279424952432677]
  Music-dance representation can be applied to three downstream tasks: (a) dance classification, (b) music-dance retrieval, and (c) music-dance. We derive the dance rhythms based on visual appearance and motion cues inspired by the music rhythm analysis. Then the visual rhythms are temporally aligned with the music counterparts, which are extracted by the amplitude of sound intensity.
  arXiv  Detail & Related papers  (2022-07-07T09:44:44Z)
• Music Gesture for Visual Sound Separation [121.36275456396075]
  "Music Gesture" is a keypoint-based structured representation to explicitly model the body and finger movements of musicians when they perform music. We first adopt a context-aware graph network to integrate visual semantic context with body dynamics, and then apply an audio-visual fusion model to associate body movements with the corresponding audio signals.
  arXiv  Detail & Related papers  (2020-04-20T17:53:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.