PRISM: Streaming Human Motion Generation with Per-Joint Latent Decomposition

• URL: http://arxiv.org/abs/2603.08590v2
• Date: Tue, 10 Mar 2026 03:27:46 GMT
• Title: PRISM: Streaming Human Motion Generation with Per-Joint Latent Decomposition
• Abstract summary: Existing motion autoencoders compress each frame into a single monolithic latent vector.<n>We present PRISM, addressing each challenge with a dedicated contribution.<n>We train a single motion generation foundation model that seamlessly handles text-to-motion, pose-conditioned generation, autoregressive sequential generation, and narrative motion composition.
• Score: 42.434871160461206
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Text-to-motion generation has advanced rapidly, yet two challenges persist. First, existing motion autoencoders compress each frame into a single monolithic latent vector, entangling trajectory and per-joint rotations in an unstructured representation that downstream generators struggle to model faithfully. Second, text-to-motion, pose-conditioned generation, and long-horizon sequential synthesis typically require separate models or task-specific mechanisms, with autoregressive approaches suffering from severe error accumulation over extended rollouts. We present PRISM, addressing each challenge with a dedicated contribution. (1) A joint-factorized motion latent space: each body joint occupies its own token, forming a structured 2D grid (time joints) compressed by a causal VAE with forward-kinematics supervision. This simple change to the latent space -- without modifying the generator -- substantially improves generation quality, revealing that latent space design has been an underestimated bottleneck. (2) Noise-free condition injection: each latent token carries its own timestep embedding, allowing conditioning frames to be injected as clean tokens (timestep0) while the remaining tokens are denoised. This unifies text-to-motion and pose-conditioned generation in a single model, and directly enables autoregressive segment chaining for streaming synthesis. Self-forcing training further suppresses drift in long rollouts. With these two components, we train a single motion generation foundation model that seamlessly handles text-to-motion, pose-conditioned generation, autoregressive sequential generation, and narrative motion composition, achieving state-of-the-art on HumanML3D, MotionHub, BABEL, and a 50-scenario user study.

Related papers

• Spatiotemporally Decoupled Autoregressive Diffusion Model for Human Motion Generation [19.887078319028102]
  We propose a unifiedtemporal decoupled framework named DeMoDiff to redesign motion representation and architecture.<n>Our framework achieves state-of-the-art reconstruction performance and compelling motion generation results.
  arXiv  Detail & Related papers  (2026-08-24T14:04:35Z)
• DrawMotion: Generating 3D Human Motions by Freehand Drawing [64.44582376360027]
  This paper introduces DrawMotion, an efficient diffusion-based framework designed for multi-condition scenarios.<n>We tackle the fine-grained motion generation task from three perspectives: 1) freehand drawing condition, 2) multi-condition fusion, and 3) training-free guidance.<n>Experiments and user studies demonstrate that the freehand drawing approach reduces user time by approximately 46.7% when generating motions aligned with their imagination.
  arXiv  Detail & Related papers  (2026-05-20T09:43:50Z)
• From Synchrony to Sequence: Exo-to-Ego Generation via Interpolation [54.68405211129937]
  Exo-Seqto-Ego video generation aims to synthesize a first-person view from a synchronized third-person and corresponding camera poses.<n> synchronized exo-ego data inherently introduces substantial-temporal discontinuities, violating the smooth-motion assumptions of standard video generation benchmarks.<n>We propose Syn2Ex-Forcing, a sequential sequence model that interpolates between the source and target videos to form a single continuous signal.
  arXiv  Detail & Related papers  (2026-04-15T12:32:25Z)
• LiveGesture Streamable Co-Speech Gesture Generation Model [15.008891901028333]
  LiveGesture is a speech-driven full-body gesture generation framework.<n>It operates with zero look-ahead and supports arbitrary sequence length.<n>It produces coherent, diverse, and beat-synchronous full-body gestures in real time.
  arXiv  Detail & Related papers  (2026-04-13T02:54:07Z)
• Coordinate-Based Dual-Constrained Autoregressive Motion Generation [28.415421434583482]
  We propose a flexible, high-fidelity, and semantically faithful text-to-motion framework, named Coordinate-based Dual-constrained Autoregressive Motion Generation (CDAMD)<n>With motion coordinates as input, CDAMD follows the autoregressive paradigm and leverages diffusion-inspired multi-layer perceptrons to enhance the fidelity of predicted motions.<n>Since there is limited work on coordinate-based motion synthesis, we establish new benchmarks for both text-to-motion generation and motion editing.
  arXiv  Detail & Related papers  (2026-04-09T11:05:29Z)
• UniMotion: A Unified Framework for Motion-Text-Vision Understanding and Generation [47.383717641500844]
  UniMotion is a framework for simultaneous understanding and generation of human motion, natural language, and RGB images.<n>It treats motion as a first-class continuous modality on equal footing with RGB.<n>UniMotion achieves state-of-the-art performance across seven tasks spanning any-to-any understanding, generation, and editing.
  arXiv  Detail & Related papers  (2026-03-23T17:59:48Z)
• Controllable Complex Human Motion Video Generation via Text-to-Skeleton Cascades [32.12940211414945]
  We propose a two-stage cascaded framework that addresses both limitations.<n>An autoregressive text-to-skeleton model generates 2D pose sequences from natural language descriptions.<n>A pose-conditioned video diffusion model synthesizes videos from a reference image and the generated skeleton sequence.
  arXiv  Detail & Related papers  (2026-03-09T07:04:29Z)
• UniHand: A Unified Model for Diverse Controlled 4D Hand Motion Modeling [45.29560152294065]
  UniHand is a unified diffusion-based framework that formulates both estimation and generation as conditional motion synthesis.<n>Visual observations are encoded with a frozen backbone, while a dedicated hand perceptron extracts hand-specific cues directly from image features.<n>A latent diffusion model then synthesizes consistent motion sequences from diverse conditions.
  arXiv  Detail & Related papers  (2026-02-25T06:53:15Z)
• HINT: Hierarchical Interaction Modeling for Autoregressive Multi-Human Motion Generation [55.73037290387896]
  We introduce HINT, the first autoregressive framework for multi-human motion generation with Hierarchical INTeraction modeling in diffusion.<n>First, HINT leverages a disentangled motion representation within a canonicalized latent space, decoupling local motion semantics from inter-person interactions.<n>Second, HINT adopts a sliding-window strategy for efficient online generation, and aggregates local within-window and global cross-window conditions to capture past human history, inter-person dependencies, and align with text guidance.
  arXiv  Detail & Related papers  (2026-01-28T08:47:23Z)
• Reward Forcing: Efficient Streaming Video Generation with Rewarded Distribution Matching Distillation [69.57572900337176]
  We introduce Reward Forcing, a novel framework for efficient streaming video generation.<n> EMA-Sink tokens capture both long-term context and recent dynamics, preventing initial frame copying.<n>Re-DMD biases the model's output distribution toward high-reward regions by prioritizing samples with greater dynamics rated by a vision-language model.
  arXiv  Detail & Related papers  (2025-12-04T11:12:13Z)
• ReCoM: Realistic Co-Speech Motion Generation with Recurrent Embedded Transformer [58.49950218437718]
  We present ReCoM, an efficient framework for generating high-fidelity and generalizable human body motions synchronized with speech.<n>The core innovation lies in the Recurrent Embedded Transformer (RET), which integrates Dynamic Embedding Regularization (DER) into a Vision Transformer (ViT) core architecture.<n>To enhance model robustness, we incorporate the proposed DER strategy, which equips the model with dual capabilities of noise resistance and cross-domain generalization.
  arXiv  Detail & Related papers  (2025-03-27T16:39:40Z)
• Dynamic Concepts Personalization from Single Videos [92.62863918003575]
  We introduce Set-and-Sequence, a novel framework for personalizing generative video models with dynamic concepts.<n>Our approach imposes a-temporal weight space within an architecture that does not explicitly separate spatial and temporal features.<n>Our framework embeds dynamic concepts into the video model's output domain, enabling unprecedented editability and compositionality.
  arXiv  Detail & Related papers  (2025-02-20T18:53:39Z)
• BAMM: Bidirectional Autoregressive Motion Model [14.668729995275807]
  Bidirectional Autoregressive Motion Model (BAMM) is a novel text-to-motion generation framework. BAMM consists of two key components: a motion tokenizer that transforms 3D human motion into discrete tokens in latent space, and a masked self-attention transformer that autoregressively predicts randomly masked tokens. This feature enables BAMM to simultaneously achieving high-quality motion generation with enhanced usability and built-in motion editability.
  arXiv  Detail & Related papers  (2024-03-28T14:04:17Z)
• Seamless Human Motion Composition with Blended Positional Encodings [38.85158088021282]
  We introduce FlowMDM, the first diffusion-based model that generates seamless Human Motion Compositions (HMC) without postprocessing or redundant denoising steps. We achieve state-of-the-art results in terms of accuracy, realism, and smoothness on the Babel and HumanML3D datasets.
  arXiv  Detail & Related papers  (2024-02-23T18:59:40Z)
• Synthesizing Long-Term Human Motions with Diffusion Models via Coherent Sampling [74.62570964142063]
  Text-to-motion generation has gained increasing attention, but most existing methods are limited to generating short-term motions. We propose a novel approach that utilizes a past-conditioned diffusion model with two optional coherent sampling methods. Our proposed method is capable of generating compositional and coherent long-term 3D human motions controlled by a user-instructed long text stream.
  arXiv  Detail & Related papers  (2023-08-03T16:18:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.