SimPoE: Simulated Character Control for 3D Human Pose Estimation
- URL: http://arxiv.org/abs/2104.00683v1
- Date: Thu, 1 Apr 2021 17:59:50 GMT
- Title: SimPoE: Simulated Character Control for 3D Human Pose Estimation
- Authors: Ye Yuan, Shih-En Wei, Tomas Simon, Kris Kitani, Jason Saragih
- Abstract summary: SimPoE is a Simulation-based approach for 3D human Pose Estimation.
It integrates image-based kinematic inference and physics-based dynamics modeling.
Our approach establishes the new state of the art in pose accuracy while ensuring physical plausibility.
- Score: 33.194787030240825
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Accurate estimation of 3D human motion from monocular video requires modeling
both kinematics (body motion without physical forces) and dynamics (motion with
physical forces). To demonstrate this, we present SimPoE, a Simulation-based
approach for 3D human Pose Estimation, which integrates image-based kinematic
inference and physics-based dynamics modeling. SimPoE learns a policy that
takes as input the current-frame pose estimate and the next image frame to
control a physically-simulated character to output the next-frame pose
estimate. The policy contains a learnable kinematic pose refinement unit that
uses 2D keypoints to iteratively refine its kinematic pose estimate of the next
frame. Based on this refined kinematic pose, the policy learns to compute
dynamics-based control (e.g., joint torques) of the character to advance the
current-frame pose estimate to the pose estimate of the next frame. This design
couples the kinematic pose refinement unit with the dynamics-based control
generation unit, which are learned jointly with reinforcement learning to
achieve accurate and physically-plausible pose estimation. Furthermore, we
propose a meta-control mechanism that dynamically adjusts the character's
dynamics parameters based on the character state to attain more accurate pose
estimates. Experiments on large-scale motion datasets demonstrate that our
approach establishes the new state of the art in pose accuracy while ensuring
physical plausibility.
Related papers
- PhysMotion: Physics-Grounded Dynamics From a Single Image [24.096925413047217]
We introduce PhysMotion, a novel framework that leverages principled physics-based simulations to guide intermediate 3D representations generated from a single image.
Our approach addresses the limitations of traditional data-driven generative models and result in more consistent physically plausible motions.
arXiv Detail & Related papers (2024-11-26T07:59:11Z) - Automated 3D Physical Simulation of Open-world Scene with Gaussian Splatting [22.40115216094332]
We present Sim Anything, a physics-based approach that endows static 3D objects with interactive dynamics.
Inspired by human visual reasoning, we propose MLLM-based Physical Property Perception.
We also simulate objects in an open-world scene with particles sampled via the Physical-Geometric Adaptive Sampling.
arXiv Detail & Related papers (2024-11-19T12:52:21Z) - Optimal-state Dynamics Estimation for Physics-based Human Motion Capture from Videos [6.093379844890164]
We propose a novel method to selectively incorporate the physics models with the kinematics observations in an online setting.
A recurrent neural network is introduced to realize a Kalman filter that attentively balances the kinematics input and simulated motion.
The proposed approach excels in the physics-based human pose estimation task and demonstrates the physical plausibility of the predictive dynamics.
arXiv Detail & Related papers (2024-10-10T10:24:59Z) - DeepSimHO: Stable Pose Estimation for Hand-Object Interaction via
Physics Simulation [81.11585774044848]
We present DeepSimHO, a novel deep-learning pipeline that combines forward physics simulation and backward gradient approximation with a neural network.
Our method noticeably improves the stability of the estimation and achieves superior efficiency over test-time optimization.
arXiv Detail & Related papers (2023-10-11T05:34:36Z) - Unsupervised 3D Pose Estimation with Non-Rigid Structure-from-Motion
Modeling [83.76377808476039]
We propose a new modeling method for human pose deformations and design an accompanying diffusion-based motion prior.
Inspired by the field of non-rigid structure-from-motion, we divide the task of reconstructing 3D human skeletons in motion into the estimation of a 3D reference skeleton.
A mixed spatial-temporal NRSfMformer is used to simultaneously estimate the 3D reference skeleton and the skeleton deformation of each frame from 2D observations sequence.
arXiv Detail & Related papers (2023-08-18T16:41:57Z) - 3D Pose Estimation and Future Motion Prediction from 2D Images [26.28886209268217]
This paper considers to jointly tackle the highly correlated tasks of estimating 3D human body poses and predicting future 3D motions from RGB image sequences.
Based on Lie algebra pose representation, a novel self-projection mechanism is proposed that naturally preserves human motion kinematics.
arXiv Detail & Related papers (2021-11-26T01:02:00Z) - Dynamics-Regulated Kinematic Policy for Egocentric Pose Estimation [23.603254270514224]
We propose a method for object-aware 3D egocentric pose estimation that tightly integrates kinematics modeling, dynamics modeling, and scene object information.
We demonstrate for the first time, the ability to estimate physically-plausible 3D human-object interactions using a single wearable camera.
arXiv Detail & Related papers (2021-06-10T17:59:50Z) - HuMoR: 3D Human Motion Model for Robust Pose Estimation [100.55369985297797]
HuMoR is a 3D Human Motion Model for Robust Estimation of temporal pose and shape.
We introduce a conditional variational autoencoder, which learns a distribution of the change in pose at each step of a motion sequence.
We demonstrate that our model generalizes to diverse motions and body shapes after training on a large motion capture dataset.
arXiv Detail & Related papers (2021-05-10T21:04:55Z) - Real-time Deep Dynamic Characters [95.5592405831368]
We propose a deep videorealistic 3D human character model displaying highly realistic shape, motion, and dynamic appearance.
We use a novel graph convolutional network architecture to enable motion-dependent deformation learning of body and clothing.
We show that our model creates motion-dependent surface deformations, physically plausible dynamic clothing deformations, as well as video-realistic surface textures at a much higher level of detail than previous state of the art approaches.
arXiv Detail & Related papers (2021-05-04T23:28:55Z) - Contact and Human Dynamics from Monocular Video [73.47466545178396]
Existing deep models predict 2D and 3D kinematic poses from video that are approximately accurate, but contain visible errors.
We present a physics-based method for inferring 3D human motion from video sequences that takes initial 2D and 3D pose estimates as input.
arXiv Detail & Related papers (2020-07-22T21:09:11Z) - Kinematic-Structure-Preserved Representation for Unsupervised 3D Human
Pose Estimation [58.72192168935338]
Generalizability of human pose estimation models developed using supervision on large-scale in-studio datasets remains questionable.
We propose a novel kinematic-structure-preserved unsupervised 3D pose estimation framework, which is not restrained by any paired or unpaired weak supervisions.
Our proposed model employs three consecutive differentiable transformations named as forward-kinematics, camera-projection and spatial-map transformation.
arXiv Detail & Related papers (2020-06-24T23:56:33Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.