WalkTheDog: Cross-Morphology Motion Alignment via Phase Manifolds

• URL: http://arxiv.org/abs/2407.18946v1
• Date: Thu, 11 Jul 2024 09:31:05 GMT
• Title: WalkTheDog: Cross-Morphology Motion Alignment via Phase Manifolds
• Authors: Peizhuo Li, Sebastian Starke, Yuting Ye, Olga Sorkine-Hornung,
• Abstract summary: We present a new approach for understanding the periodicity structure and semantics of motion datasets. We learn a shared phase manifold for multiple characters, such as a human and a dog, without any supervision. In combination with an improved motion matching framework, we demonstrate the manifold's capability of timing and semantics alignment in several applications.
• Score: 23.884105024013714
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present a new approach for understanding the periodicity structure and semantics of motion datasets, independently of the morphology and skeletal structure of characters. Unlike existing methods using an overly sparse high-dimensional latent, we propose a phase manifold consisting of multiple closed curves, each corresponding to a latent amplitude. With our proposed vector quantized periodic autoencoder, we learn a shared phase manifold for multiple characters, such as a human and a dog, without any supervision. This is achieved by exploiting the discrete structure and a shallow network as bottlenecks, such that semantically similar motions are clustered into the same curve of the manifold, and the motions within the same component are aligned temporally by the phase variable. In combination with an improved motion matching framework, we demonstrate the manifold's capability of timing and semantics alignment in several applications, including motion retrieval, transfer and stylization. Code and pre-trained models for this paper are available at https://peizhuoli.github.io/walkthedog.

Related papers

• FunPhase: A Periodic Functional Autoencoder for Motion Generation via Phase Manifolds [2.6041136107390037]
  We introduce FunPhase, a functional periodic autoencoder that learns a phase manifold for motion and replaces discrete temporal decoding with a function-space formulation.<n>FunPhase supports downstream tasks such as super-resolution and partial-body motion completion, generalizes across skeletons and datasets, and unifies motion prediction and generation within a single interpretable manifold.
  arXiv  Detail & Related papers  (2025-12-10T08:46:53Z)
• Modality Alignment across Trees on Heterogeneous Hyperbolic Manifolds [49.95082206008502]
  Alignment across Trees is a method that constructs and aligns tree-like hierarchical features for both image and text modalities.<n>We introduce a semantic-aware visual feature extraction framework that applies a cross-attention mechanism to visual class tokens from intermediate Transformer layers.
  arXiv  Detail & Related papers  (2025-10-31T11:32:15Z)
• Kuramoto Orientation Diffusion Models [67.0711709825854]
  Orientation-rich images, such as fingerprints and textures, often exhibit coherent angular patterns.<n>Motivated by the role of phase synchronization in biological systems, we propose a score-based generative model.<n>We implement competitive results on general image benchmarks and significantly improves generation quality on orientation-dense datasets like fingerprints and textures.
  arXiv  Detail & Related papers  (2025-09-18T18:18:49Z)
• Real-Time Moving Flock Detection in Pedestrian Trajectories Using Sequential Deep Learning Models [1.2289361708127877]
  This paper investigates the use of sequential deep learning models, including Recurrent Neural Networks (RNNs), for real-time flock detection in multi-pedestrian trajectories. We validate our method using real-world group movement datasets, demonstrating its robustness across varying sequence lengths and diverse movement patterns. We extend our approach to identify other forms of collective motion, such as convoys and swarms, paving the way for more comprehensive multi-agent behavior analysis.
  arXiv  Detail & Related papers  (2025-02-21T07:04:34Z)
• Technical Report: Masked Skeleton Sequence Modeling for Learning Larval Zebrafish Behavior Latent Embeddings [5.922172844641853]
  We introduce a novel self-supervised learning method for extracting latent embeddings from behaviors of larval zebrafish. For the skeletal sequences of swimming zebrafish, we propose a pioneering Transformer-CNN architecture, the Sequence Spatial-Temporal Transformer (SSTFormer)
  arXiv  Detail & Related papers  (2024-03-23T02:58:10Z)
• Generative Motion Stylization of Cross-structure Characters within Canonical Motion Space [28.628241993271647]
  We propose a generative motion stylization pipeline, named MotionS, for diverse and stylized motion on cross-structure characters. Our key insight is to embed motion style into a cross-modality latent space, allowing for motion stylization within a canonical motion space.
  arXiv  Detail & Related papers  (2024-03-18T04:41:59Z)
• A Decoupled Spatio-Temporal Framework for Skeleton-based Action Segmentation [89.86345494602642]
  Existing methods are limited in weak-temporal modeling capability. We propose a Decoupled Scoupled Framework (DeST) to address the issues. DeST significantly outperforms current state-of-the-art methods with less computational complexity.
  arXiv  Detail & Related papers  (2023-12-10T09:11:39Z)
• Motion In-Betweening with Phase Manifolds [29.673541655825332]
  This paper introduces a novel data-driven motion in-betweening system to reach target poses of characters by making use of phases variables learned by a Periodic Autoencoder. Our approach utilizes a mixture-of-experts neural network model, in which the phases cluster movements in both space and time with different expert weights.
  arXiv  Detail & Related papers  (2023-08-24T12:56:39Z)
• Learning the Evolutionary and Multi-scale Graph Structure for Multivariate Time Series Forecasting [50.901984244738806]
  We show how to model the evolutionary and multi-scale interactions of time series. In particular, we first provide a hierarchical graph structure cooperated with the dilated convolution to capture the scale-specific correlations. A unified neural network is provided to integrate the components above to get the final prediction.
  arXiv  Detail & Related papers  (2022-06-28T08:11:12Z)
• MoDi: Unconditional Motion Synthesis from Diverse Data [51.676055380546494]
  We present MoDi, an unconditional generative model that synthesizes diverse motions. Our model is trained in a completely unsupervised setting from a diverse, unstructured and unlabeled motion dataset. We show that despite the lack of any structure in the dataset, the latent space can be semantically clustered.
  arXiv  Detail & Related papers  (2022-06-16T09:06:25Z)
• MultiBodySync: Multi-Body Segmentation and Motion Estimation via 3D Scan Synchronization [61.015704878681795]
  We present a novel, end-to-end trainable multi-body motion segmentation and rigid registration framework for 3D point clouds. The two non-trivial challenges posed by this multi-scan multibody setting are. guaranteeing correspondence and segmentation consistency across multiple input point clouds and. obtaining robust motion-based rigid body segmentation applicable to novel object categories.
  arXiv  Detail & Related papers  (2021-01-17T06:36:28Z)
• Hierarchical Style-based Networks for Motion Synthesis [150.226137503563]
  We propose a self-supervised method for generating long-range, diverse and plausible behaviors to achieve a specific goal location. Our proposed method learns to model the motion of human by decomposing a long-range generation task in a hierarchical manner. On large-scale skeleton dataset, we show that the proposed method is able to synthesise long-range, diverse and plausible motion.
  arXiv  Detail & Related papers  (2020-08-24T02:11:02Z)
• Skeleton-Aware Networks for Deep Motion Retargeting [83.65593033474384]
  We introduce a novel deep learning framework for data-driven motion between skeletons. Our approach learns how to retarget without requiring any explicit pairing between the motions in the training set.
  arXiv  Detail & Related papers  (2020-05-12T12:51:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.