Neural Marionette: Unsupervised Learning of Motion Skeleton and Latent Dynamics from Volumetric Video

• URL: http://arxiv.org/abs/2202.08418v1
• Date: Thu, 17 Feb 2022 02:44:16 GMT
• Title: Neural Marionette: Unsupervised Learning of Motion Skeleton and Latent Dynamics from Volumetric Video
• Authors: Jinseok Bae, Hojun Jang, Cheol-Hui Min, Hyungun Choi, Young Min Kim
• Abstract summary: We present Neural Marionette, an unsupervised approach that discovers the skeletal structure from a dynamic sequence. We demonstrate that the discovered structure is even comparable to the hand-labeled ground truth in skeleton representing a 4D sequence of motion.
• Score: 5.456297943378056
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present Neural Marionette, an unsupervised approach that discovers the skeletal structure from a dynamic sequence and learns to generate diverse motions that are consistent with the observed motion dynamics. Given a video stream of point cloud observation of an articulated body under arbitrary motion, our approach discovers the unknown low-dimensional skeletal relationship that can effectively represent the movement. Then the discovered structure is utilized to encode the motion priors of dynamic sequences in a latent structure, which can be decoded to the relative joint rotations to represent the full skeletal motion. Our approach works without any prior knowledge of the underlying motion or skeletal structure, and we demonstrate that the discovered structure is even comparable to the hand-labeled ground truth skeleton in representing a 4D sequence of motion. The skeletal structure embeds the general semantics of possible motion space that can generate motions for diverse scenarios. We verify that the learned motion prior is generalizable to the multi-modal sequence generation, interpolation of two poses, and motion retargeting to a different skeletal structure.

Related papers

• PALUM: Part-based Attention Learning for Unified Motion Retargeting [53.17113525688095]
  Remotion between characters with different skeleton structures is a fundamental challenge in computer animation.<n>We present a novel approach that learns common motion representations across diverse skeleton topologies.<n>Experiments demonstrate superior performance in handling diverse skeletal structures while maintaining motion realism and semantic fidelity.
  arXiv  Detail & Related papers  (2026-01-12T07:29:44Z)
• Topology-Agnostic Animal Motion Generation from Text Prompt [16.557163253248817]
  We introduce OmniZoo, a large-scale animal motion dataset spanning 140 species and 32,979 sequences.<n>We propose a generalized autoregressive motion generation framework capable of producing text-driven motions for arbitrary skeletal topologies.
  arXiv  Detail & Related papers  (2025-12-11T07:08:29Z)
• AnyTop: Character Animation Diffusion with Any Topology [54.07731933876742]
  We introduce AnyTop, a diffusion model that generates motions for diverse characters with distinct motion dynamics. Our work features a transformer-based denoising network, tailored for arbitrary skeleton learning. Our evaluation demonstrates that AnyTops well, even with as few as three training examples per topology, and can produce motions for unseen skeletons as well.
  arXiv  Detail & Related papers  (2025-02-24T17:00:36Z)
• Evolving Skeletons: Motion Dynamics in Action Recognition [2.290956583394892]
  We compare skeletal graph and hypergraph representations, analyzing static poses against motion-injected poses. Our findings highlight the strengths and limitations of Taylor-transformed skeletons, demonstrating their potential to enhance motion dynamics. This study underscores the need for innovative skeletal modelling techniques to handle motion-rich data and advance the field of action recognition.
  arXiv  Detail & Related papers  (2025-01-05T16:16:10Z)
• Neuron: Learning Context-Aware Evolving Representations for Zero-Shot Skeleton Action Recognition [64.56321246196859]
  We propose a novel dyNamically Evolving dUal skeleton-semantic syneRgistic framework. We first construct the spatial-temporal evolving micro-prototypes and integrate dynamic context-aware side information. We introduce the spatial compression and temporal memory mechanisms to guide the growth of spatial-temporal micro-prototypes.
  arXiv  Detail & Related papers  (2024-11-18T05:16:11Z)
• Shape of Motion: 4D Reconstruction from a Single Video [51.04575075620677]
  We introduce a method capable of reconstructing generic dynamic scenes, featuring explicit, full-sequence-long 3D motion. We exploit the low-dimensional structure of 3D motion by representing scene motion with a compact set of SE3 motion bases. Our method achieves state-of-the-art performance for both long-range 3D/2D motion estimation and novel view synthesis on dynamic scenes.
  arXiv  Detail & Related papers  (2024-07-18T17:59:08Z)
• Generative Motion Stylization of Cross-structure Characters within Canonical Motion Space [28.628241993271647]
  We propose a generative motion stylization pipeline, named MotionS, for diverse and stylized motion on cross-structure characters. Our key insight is to embed motion style into a cross-modality latent space, allowing for motion stylization within a canonical motion space.
  arXiv  Detail & Related papers  (2024-03-18T04:41:59Z)
• MoDi: Unconditional Motion Synthesis from Diverse Data [51.676055380546494]
  We present MoDi, an unconditional generative model that synthesizes diverse motions. Our model is trained in a completely unsupervised setting from a diverse, unstructured and unlabeled motion dataset. We show that despite the lack of any structure in the dataset, the latent space can be semantically clustered.
  arXiv  Detail & Related papers  (2022-06-16T09:06:25Z)
• GANimator: Neural Motion Synthesis from a Single Sequence [38.361579401046875]
  We present GANimator, a generative model that learns to synthesize novel motions from a single, short motion sequence. GANimator generates motions that resemble the core elements of the original motion, while simultaneously synthesizing novel and diverse movements. We show a number of applications, including crowd simulation, key-frame editing, style transfer, and interactive control, which all learn from a single input sequence.
  arXiv  Detail & Related papers  (2022-05-05T13:04:14Z)
• MoCaNet: Motion Retargeting in-the-wild via Canonicalization Networks [77.56526918859345]
  We present a novel framework that brings the 3D motion task from controlled environments to in-the-wild scenarios. It is capable of body motion from a character in a 2D monocular video to a 3D character without using any motion capture system or 3D reconstruction procedure.
  arXiv  Detail & Related papers  (2021-12-19T07:52:05Z)
• A Hierarchy-Aware Pose Representation for Deep Character Animation [2.47343886645587]
  We present a robust pose representation for motion modeling, suitable for deep character animation. Our representation is based on dual quaternions, the mathematical abstractions with well-defined operations, which simultaneously encode rotational and positional orientation. We show that our representation overcomes common motion artifacts, and assess its performance compared to other popular representations.
  arXiv  Detail & Related papers  (2021-11-27T14:33:24Z)
• Hierarchical Style-based Networks for Motion Synthesis [150.226137503563]
  We propose a self-supervised method for generating long-range, diverse and plausible behaviors to achieve a specific goal location. Our proposed method learns to model the motion of human by decomposing a long-range generation task in a hierarchical manner. On large-scale skeleton dataset, we show that the proposed method is able to synthesise long-range, diverse and plausible motion.
  arXiv  Detail & Related papers  (2020-08-24T02:11:02Z)
• Skeleton-Aware Networks for Deep Motion Retargeting [83.65593033474384]
  We introduce a novel deep learning framework for data-driven motion between skeletons. Our approach learns how to retarget without requiring any explicit pairing between the motions in the training set.
  arXiv  Detail & Related papers  (2020-05-12T12:51:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.