Within the Dynamic Context: Inertia-aware 3D Human Modeling with Pose Sequence

• URL: http://arxiv.org/abs/2403.19160v2
• Date: Tue, 16 Jul 2024 06:12:46 GMT
• Title: Within the Dynamic Context: Inertia-aware 3D Human Modeling with Pose Sequence
• Authors: Yutong Chen, Yifan Zhan, Zhihang Zhong, Wei Wang, Xiao Sun, Yu Qiao, Yinqiang Zheng,
• Abstract summary: In this study, we elucidate that variations in human appearance depend not only on the current frame's pose condition but also on past pose states. We introduce Dyco, a novel method utilizing the delta pose sequence representation for non-rigid deformations. In addition, our inertia-aware 3D human method can unprecedentedly simulate appearance changes caused by inertia at different velocities.
• Score: 47.16903508897047
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural rendering techniques have significantly advanced 3D human body modeling. However, previous approaches often overlook dynamics induced by factors such as motion inertia, leading to challenges in scenarios like abrupt stops after rotation, where the pose remains static while the appearance changes. This limitation arises from reliance on a single pose as conditional input, resulting in ambiguity in mapping one pose to multiple appearances. In this study, we elucidate that variations in human appearance depend not only on the current frame's pose condition but also on past pose states. Therefore, we introduce Dyco, a novel method utilizing the delta pose sequence representation for non-rigid deformations and canonical space to effectively model temporal appearance variations. To prevent a decrease in the model's generalization ability to novel poses, we further propose low-dimensional global context to reduce unnecessary inter-body part dependencies and a quantization operation to mitigate overfitting of the delta pose sequence by the model. To validate the effectiveness of our approach, we collected a novel dataset named I3D-Human, with a focus on capturing temporal changes in clothing appearance under approximate poses. Through extensive experiments on both I3D-Human and existing datasets, our approach demonstrates superior qualitative and quantitative performance. In addition, our inertia-aware 3D human method can unprecedentedly simulate appearance changes caused by inertia at different velocities.

Related papers

• Unsupervised 3D Pose Estimation with Non-Rigid Structure-from-Motion Modeling [83.76377808476039]
  We propose a new modeling method for human pose deformations and design an accompanying diffusion-based motion prior. Inspired by the field of non-rigid structure-from-motion, we divide the task of reconstructing 3D human skeletons in motion into the estimation of a 3D reference skeleton. A mixed spatial-temporal NRSfMformer is used to simultaneously estimate the 3D reference skeleton and the skeleton deformation of each frame from 2D observations sequence.
  arXiv  Detail & Related papers  (2023-08-18T16:41:57Z)
• Pose-NDF: Modeling Human Pose Manifolds with Neural Distance Fields [47.62275563070933]
  We present a continuous model for plausible human poses based on neural distance fields (NDFs) Pose-NDF learns a manifold of plausible poses as the zero level set of a neural implicit function. It can be used to generate more diverse poses by random sampling and projection than VAE-based methods.
  arXiv  Detail & Related papers  (2022-07-27T21:46:47Z)
• Learning Motion-Dependent Appearance for High-Fidelity Rendering of Dynamic Humans from a Single Camera [49.357174195542854]
  A key challenge of learning the dynamics of the appearance lies in the requirement of a prohibitively large amount of observations. We show that our method can generate a temporally coherent video of dynamic humans for unseen body poses and novel views given a single view video.
  arXiv  Detail & Related papers  (2022-03-24T00:22:03Z)
• 3D Skeleton-based Human Motion Prediction with Manifold-Aware GAN [3.1313293632309827]
  We propose a novel solution for 3D skeleton-based human motion prediction. We build a manifold-aware Wasserstein generative adversarial model that captures the temporal and spatial dependencies of human motion. Experiments have been conducted on CMU MoCap and Human 3.6M datasets.
  arXiv  Detail & Related papers  (2022-03-01T20:49:13Z)
• LatentHuman: Shape-and-Pose Disentangled Latent Representation for Human Bodies [78.17425779503047]
  We propose a novel neural implicit representation for the human body. It is fully differentiable and optimizable with disentangled shape and pose latent spaces. Our model can be trained and fine-tuned directly on non-watertight raw data with well-designed losses.
  arXiv  Detail & Related papers  (2021-11-30T04:10:57Z)
• HuMoR: 3D Human Motion Model for Robust Pose Estimation [100.55369985297797]
  HuMoR is a 3D Human Motion Model for Robust Estimation of temporal pose and shape. We introduce a conditional variational autoencoder, which learns a distribution of the change in pose at each step of a motion sequence. We demonstrate that our model generalizes to diverse motions and body shapes after training on a large motion capture dataset.
  arXiv  Detail & Related papers  (2021-05-10T21:04:55Z)
• Kinematic-Structure-Preserved Representation for Unsupervised 3D Human Pose Estimation [58.72192168935338]
  Generalizability of human pose estimation models developed using supervision on large-scale in-studio datasets remains questionable. We propose a novel kinematic-structure-preserved unsupervised 3D pose estimation framework, which is not restrained by any paired or unpaired weak supervisions. Our proposed model employs three consecutive differentiable transformations named as forward-kinematics, camera-projection and spatial-map transformation.
  arXiv  Detail & Related papers  (2020-06-24T23:56:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.