BoDiffusion: Diffusing Sparse Observations for Full-Body Human Motion Synthesis

• URL: http://arxiv.org/abs/2304.11118v1
• Date: Fri, 21 Apr 2023 16:39:05 GMT
• Title: BoDiffusion: Diffusing Sparse Observations for Full-Body Human Motion Synthesis
• Authors: Angela Castillo, Maria Escobar, Guillaume Jeanneret, Albert Pumarola, Pablo Arbel\'aez, Ali Thabet, Artsiom Sanakoyeu
• Abstract summary: Mixed reality applications require tracking the user's full-body motion to enable an immersive experience. We propose BoDiffusion -- a generative diffusion model for motion synthesis to tackle this under-constrained reconstruction problem. We present a time and space conditioning scheme that allows BoDiffusion to leverage sparse tracking inputs while generating smooth and realistic full-body motion sequences.
• Score: 14.331548412833513
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Mixed reality applications require tracking the user's full-body motion to enable an immersive experience. However, typical head-mounted devices can only track head and hand movements, leading to a limited reconstruction of full-body motion due to variability in lower body configurations. We propose BoDiffusion -- a generative diffusion model for motion synthesis to tackle this under-constrained reconstruction problem. We present a time and space conditioning scheme that allows BoDiffusion to leverage sparse tracking inputs while generating smooth and realistic full-body motion sequences. To the best of our knowledge, this is the first approach that uses the reverse diffusion process to model full-body tracking as a conditional sequence generation task. We conduct experiments on the large-scale motion-capture dataset AMASS and show that our approach outperforms the state-of-the-art approaches by a significant margin in terms of full-body motion realism and joint reconstruction error.

Related papers

• COIN: Control-Inpainting Diffusion Prior for Human and Camera Motion Estimation [98.05046790227561]
  COIN is a control-inpainting motion diffusion prior that enables fine-grained control to disentangle human and camera motions. COIN outperforms the state-of-the-art methods in terms of global human motion estimation and camera motion estimation.
  arXiv  Detail & Related papers  (2024-08-29T10:36:29Z)
• Stratified Avatar Generation from Sparse Observations [10.291918304187769]
  Estimating 3D full-body avatars from AR/VR devices is essential for creating immersive experiences. In this paper, we are inspired by the inherent property of the kinematic tree defined in the Skinned Multi-Person Linear (SMPL) model. We propose a stratified approach to decouple the conventional full-body avatar reconstruction pipeline into two stages.
  arXiv  Detail & Related papers  (2024-05-30T06:25:42Z)
• MotionMix: Weakly-Supervised Diffusion for Controllable Motion Generation [19.999239668765885]
  MotionMix is a weakly-supervised diffusion model that leverages both noisy and unannotated motion sequences. Our framework consistently achieves state-of-the-art performances on text-to-motion, action-to-motion, and music-to-dance tasks.
  arXiv  Detail & Related papers  (2024-01-20T04:58:06Z)
• RoHM: Robust Human Motion Reconstruction via Diffusion [58.63706638272891]
  RoHM is an approach for robust 3D human motion reconstruction from monocular RGB(-D) videos. It conditioned on noisy and occluded input data, reconstructs complete, plausible motions in consistent global coordinates. Our method outperforms state-of-the-art approaches qualitatively and quantitatively, while being faster at test time.
  arXiv  Detail & Related papers  (2024-01-16T18:57:50Z)
• Motion Flow Matching for Human Motion Synthesis and Editing [75.13665467944314]
  We propose emphMotion Flow Matching, a novel generative model for human motion generation featuring efficient sampling and effectiveness in motion editing applications. Our method reduces the sampling complexity from thousand steps in previous diffusion models to just ten steps, while achieving comparable performance in text-to-motion and action-to-motion generation benchmarks.
  arXiv  Detail & Related papers  (2023-12-14T12:57:35Z)
• Avatars Grow Legs: Generating Smooth Human Motion from Sparse Tracking Inputs with Diffusion Model [18.139630622759636]
  We present AGRoL, a novel conditional diffusion model specifically designed to track full bodies given sparse upper-body tracking signals. Our model is based on a simple multi-layer perceptron (MLP) architecture and a novel conditioning scheme for motion data. Unlike common diffusion architectures, our compact architecture can run in real-time, making it suitable for online body-tracking applications.
  arXiv  Detail & Related papers  (2023-04-17T19:35:13Z)
• Executing your Commands via Motion Diffusion in Latent Space [51.64652463205012]
  We propose a Motion Latent-based Diffusion model (MLD) to produce vivid motion sequences conforming to the given conditional inputs. Our MLD achieves significant improvements over the state-of-the-art methods among extensive human motion generation tasks.
  arXiv  Detail & Related papers  (2022-12-08T03:07:00Z)
• PhysDiff: Physics-Guided Human Motion Diffusion Model [101.1823574561535]
  Existing motion diffusion models largely disregard the laws of physics in the diffusion process. PhysDiff incorporates physical constraints into the diffusion process. Our approach achieves state-of-the-art motion quality and improves physical plausibility drastically.
  arXiv  Detail & Related papers  (2022-12-05T18:59:52Z)
• Transformer Inertial Poser: Attention-based Real-time Human Motion Reconstruction from Sparse IMUs [79.72586714047199]
  We propose an attention-based deep learning method to reconstruct full-body motion from six IMU sensors in real-time. Our method achieves new state-of-the-art results both quantitatively and qualitatively, while being simple to implement and smaller in size.
  arXiv  Detail & Related papers  (2022-03-29T16:24:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.