A Unified Masked Autoencoder with Patchified Skeletons for Motion Synthesis

• URL: http://arxiv.org/abs/2308.07301v2
• Date: Mon, 8 Apr 2024 15:47:20 GMT
• Title: A Unified Masked Autoencoder with Patchified Skeletons for Motion Synthesis
• Authors: Esteve Valls Mascaro, Hyemin Ahn, Dongheui Lee,
• Abstract summary: We present a novel task-independent model called UNIMASK-M, which can effectively address challenges using a unified architecture. Inspired by Vision TransformersVi (Ts), our UNIMASK-M model decomposes a human pose into body parts to leverage thetemporal relationships existing in human motion. Experimental results show that our model successfully forecasts human motion on the Human3.6M dataset.
• Score: 14.347147051922175
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The synthesis of human motion has traditionally been addressed through task-dependent models that focus on specific challenges, such as predicting future motions or filling in intermediate poses conditioned on known key-poses. In this paper, we present a novel task-independent model called UNIMASK-M, which can effectively address these challenges using a unified architecture. Our model obtains comparable or better performance than the state-of-the-art in each field. Inspired by Vision Transformers (ViTs), our UNIMASK-M model decomposes a human pose into body parts to leverage the spatio-temporal relationships existing in human motion. Moreover, we reformulate various pose-conditioned motion synthesis tasks as a reconstruction problem with different masking patterns given as input. By explicitly informing our model about the masked joints, our UNIMASK-M becomes more robust to occlusions. Experimental results show that our model successfully forecasts human motion on the Human3.6M dataset. Moreover, it achieves state-of-the-art results in motion inbetweening on the LaFAN1 dataset, particularly in long transition periods. More information can be found on the project website https://evm7.github.io/UNIMASKM-page/

Related papers

• Diffgrasp: Whole-Body Grasping Synthesis Guided by Object Motion Using a Diffusion Model [25.00532805042292]
  We propose a simple yet effective framework that jointly models the relationship between the body, hands, and the given object motion sequences. We introduce novel contact-aware losses and incorporate a data-driven, carefully designed guidance. Experimental results demonstrate that our approach outperforms the state-of-the-art method and generates plausible whole-body motion sequences.
  arXiv  Detail & Related papers  (2024-12-30T02:21:43Z)
• HUMOS: Human Motion Model Conditioned on Body Shape [54.20419874234214]
  We introduce a new approach to develop a generative motion model based on body shape. We show that it's possible to train this model using unpaired data. The resulting model generates diverse, physically plausible, and dynamically stable human motions.
  arXiv  Detail & Related papers  (2024-09-05T23:50:57Z)
• Scaling Up Dynamic Human-Scene Interaction Modeling [58.032368564071895]
  TRUMANS is the most comprehensive motion-captured HSI dataset currently available. It intricately captures whole-body human motions and part-level object dynamics. We devise a diffusion-based autoregressive model that efficiently generates HSI sequences of any length.
  arXiv  Detail & Related papers  (2024-03-13T15:45:04Z)
• SoMoFormer: Multi-Person Pose Forecasting with Transformers [15.617263162155062]
  We present a new method, called Social Motion Transformer (SoMoFormer), for multi-person 3D pose forecasting. Our transformer architecture uniquely models human motion input as a joint sequence rather than a time sequence. We show that with this problem reformulation, SoMoFormer naturally extends to multi-person scenes by using the joints of all people in a scene as input queries.
  arXiv  Detail & Related papers  (2022-08-30T06:59:28Z)
• MoDi: Unconditional Motion Synthesis from Diverse Data [51.676055380546494]
  We present MoDi, an unconditional generative model that synthesizes diverse motions. Our model is trained in a completely unsupervised setting from a diverse, unstructured and unlabeled motion dataset. We show that despite the lack of any structure in the dataset, the latent space can be semantically clustered.
  arXiv  Detail & Related papers  (2022-06-16T09:06:25Z)
• TEMOS: Generating diverse human motions from textual descriptions [53.85978336198444]
  We address the problem of generating diverse 3D human motions from textual descriptions. We propose TEMOS, a text-conditioned generative model leveraging variational autoencoder (VAE) training with human motion data. We show that TEMOS framework can produce both skeleton-based animations as in prior work, as well more expressive SMPL body motions.
  arXiv  Detail & Related papers  (2022-04-25T14:53:06Z)
• Transformer Inertial Poser: Attention-based Real-time Human Motion Reconstruction from Sparse IMUs [79.72586714047199]
  We propose an attention-based deep learning method to reconstruct full-body motion from six IMU sensors in real-time. Our method achieves new state-of-the-art results both quantitatively and qualitatively, while being simple to implement and smaller in size.
  arXiv  Detail & Related papers  (2022-03-29T16:24:52Z)
• Motion Prediction via Joint Dependency Modeling in Phase Space [40.54430409142653]
  We introduce a novel convolutional neural model to leverage explicit prior knowledge of motion anatomy. We then propose a global optimization module that learns the implicit relationships between individual joint features. Our method is evaluated on large-scale 3D human motion benchmark datasets.
  arXiv  Detail & Related papers  (2022-01-07T08:30:01Z)
• Learning Local Recurrent Models for Human Mesh Recovery [50.85467243778406]
  We present a new method for video mesh recovery that divides the human mesh into several local parts following the standard skeletal model. We then model the dynamics of each local part with separate recurrent models, with each model conditioned appropriately based on the known kinematic structure of the human body. This results in a structure-informed local recurrent learning architecture that can be trained in an end-to-end fashion with available annotations.
  arXiv  Detail & Related papers  (2021-07-27T14:30:33Z)
• Task-Generic Hierarchical Human Motion Prior using VAEs [44.356707509079044]
  A deep generative model that describes human motions can benefit a wide range of fundamental computer vision and graphics tasks. We present a method for learning complex human motions independent of specific tasks using a combined global and local latent space. We demonstrate the effectiveness of our hierarchical motion variational autoencoder in a variety of tasks including video-based human pose estimation.
  arXiv  Detail & Related papers  (2021-06-07T23:11:42Z)
• Graph-based Normalizing Flow for Human Motion Generation and Reconstruction [20.454140530081183]
  We propose a probabilistic generative model to synthesize and reconstruct long horizon motion sequences conditioned on past information and control signals. We evaluate the models on a mixture of motion capture datasets of human locomotion with foot-step and bone-length analysis.
  arXiv  Detail & Related papers  (2021-04-07T09:51:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.