Controllable Complex Human Motion Video Generation via Text-to-Skeleton Cascades

• URL: http://arxiv.org/abs/2603.08028v1
• Date: Mon, 09 Mar 2026 07:04:29 GMT
• Title: Controllable Complex Human Motion Video Generation via Text-to-Skeleton Cascades
• Abstract summary: We propose a two-stage cascaded framework that addresses both limitations.<n>An autoregressive text-to-skeleton model generates 2D pose sequences from natural language descriptions.<n>A pose-conditioned video diffusion model synthesizes videos from a reference image and the generated skeleton sequence.
• Score: 32.12940211414945
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Generating videos of complex human motions such as flips, cartwheels, and martial arts remains challenging for current video diffusion models. Text-only conditioning is temporally ambiguous for fine-grained motion control, while explicit pose-based controls, though effective, require users to provide complete skeleton sequences that are costly to produce for long and dynamic actions. We propose a two-stage cascaded framework that addresses both limitations. First, an autoregressive text-to-skeleton model generates 2D pose sequences from natural language descriptions by predicting each joint conditioned on previously generated poses. This design captures long-range temporal dependencies and inter-joint coordination required for complex motions. Second, a pose-conditioned video diffusion model synthesizes videos from a reference image and the generated skeleton sequence. It employs DINO-ALF (Adaptive Layer Fusion), a multi-level reference encoder that preserves appearance and clothing details under large pose changes and self-occlusions. To address the lack of publicly available datasets for complex human motion video generation, we introduce a Blender-based synthetic dataset containing 2,000 videos with diverse characters performing acrobatic and stunt-like motions. The dataset provides full control over appearance, motion, and environment. It fills an important gap because existing benchmarks significantly under-represent acrobatic motions while web-collected datasets raise copyright and privacy concerns. Experiments on our synthetic dataset and the Motion-X Fitness benchmark show that our text-to-skeleton model outperforms prior methods on FID, R-precision, and motion diversity. Our pose-to-video model also achieves the best results among all compared methods on VBench metrics for temporal consistency, motion smoothness, and subject preservation.

Related papers

• SparseCtrl-HOI: Sparse Temporal Control for Human-Object Interaction Video Generation [29.683204938136758]
  Human-Object Interaction generation aims to synthesize realistic humans manipulating diverse objects.<n>Existing approaches to this problem rely on dense temporal guidance.<n>We introduce SparseCtrl-HOI, a novel sparse temporal control framework for HOI video generation.
  arXiv  Detail & Related papers  (2026-07-07T08:29:25Z)
• DynaVid: Learning to Generate Highly Dynamic Videos using Synthetic Motion Data [51.316274891736164]
  DynaVid is a video synthesis framework that leverages synthetic motion data in training.<n>We show that DynaVid improves the realism and controllability in dynamic motion generation and camera motion control.
  arXiv  Detail & Related papers  (2026-04-02T06:12:38Z)
• Kimodo: Scaling Controllable Human Motion Generation [77.66868439601062]
  Kimodo is a controllable kinematic motion diffusion model trained on 700 hours of optical motion capture data.<n>Our model generates high-quality motions while being easily controlled through text and a comprehensive suite of kinematic constraints.
  arXiv  Detail & Related papers  (2026-03-16T17:09:30Z)
• EchoMotion: Unified Human Video and Motion Generation via Dual-Modality Diffusion Transformer [64.69014756863331]
  We introduce EchoMotion, a framework designed to model the joint distribution of appearance and human motion.<n>We also propose MVS-RoPE, which offers unified 3D positional encoding for both video and motion tokens.<n>Our findings reveal that explicitly representing human motion is to appearance, significantly boosting the coherence and plausibility of human-centric video generation.
  arXiv  Detail & Related papers  (2025-12-21T17:08:14Z)
• UniMo: Unifying 2D Video and 3D Human Motion with an Autoregressive Framework [54.337290937468175]
  We propose UniMo, an autoregressive model for joint modeling of 2D human videos and 3D human motions within a unified framework.<n>We show that our method simultaneously generates corresponding videos and motions while performing accurate motion capture.
  arXiv  Detail & Related papers  (2025-12-03T16:03:18Z)
• SynMotion: Semantic-Visual Adaptation for Motion Customized Video Generation [56.90807453045657]
  SynMotion is a motion-customized video generation model that jointly leverages semantic guidance and visual adaptation.<n>At the semantic level, we introduce the dual-em semantic comprehension mechanism which disentangles subject and motion representations.<n>At the visual level, we integrate efficient motion adapters into a pre-trained video generation model to enhance motion fidelity and temporal coherence.
  arXiv  Detail & Related papers  (2025-06-30T10:09:32Z)
• LatentMove: Towards Complex Human Movement Video Generation [35.83863053692456]
  We present LatentMove, a DiT-based framework specifically tailored for highly dynamic human animation.<n>Our architecture incorporates a conditional control branch and learnable face/body tokens to preserve consistency as well as fine-grained details across frames.<n>We introduce Complex-Human-Videos (CHV), a dataset featuring diverse, challenging human motions designed to benchmark the robustness of I2V systems.
  arXiv  Detail & Related papers  (2025-05-28T07:10:49Z)
• LEO: Generative Latent Image Animator for Human Video Synthesis [38.99490968487773]
  We propose a novel framework for human video synthesis, placing emphasis on synthesizing-temporal coherency. Our key idea is to represent motion as a sequence of flow maps in the generation process, which inherently isolate motion from appearance. We implement this idea via a flow-based image animator and a Latent Motion Diffusion Model (LMDM)
  arXiv  Detail & Related papers  (2023-05-06T09:29:12Z)
• Render In-between: Motion Guided Video Synthesis for Action Interpolation [53.43607872972194]
  We propose a motion-guided frame-upsampling framework that is capable of producing realistic human motion and appearance. A novel motion model is trained to inference the non-linear skeletal motion between frames by leveraging a large-scale motion-capture dataset. Our pipeline only requires low-frame-rate videos and unpaired human motion data but does not require high-frame-rate videos for training.
  arXiv  Detail & Related papers  (2021-11-01T15:32:51Z)
• Hierarchical Style-based Networks for Motion Synthesis [150.226137503563]
  We propose a self-supervised method for generating long-range, diverse and plausible behaviors to achieve a specific goal location. Our proposed method learns to model the motion of human by decomposing a long-range generation task in a hierarchical manner. On large-scale skeleton dataset, we show that the proposed method is able to synthesise long-range, diverse and plausible motion.
  arXiv  Detail & Related papers  (2020-08-24T02:11:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.