PC-HMR: Pose Calibration for 3D Human Mesh Recovery from 2D Images/Videos

• URL: http://arxiv.org/abs/2103.09009v2
• Date: Thu, 18 Mar 2021 17:13:37 GMT
• Title: PC-HMR: Pose Calibration for 3D Human Mesh Recovery from 2D Images/Videos
• Authors: Tianyu Luan, Yali Wang, Junhao Zhang, Zhe Wang, Zhipeng Zhou, Yu Qiao
• Abstract summary: We develop two novel Pose frameworks, i.e., Serial PC-HMR and Parallel PC-HMR. Our frameworks are based on generic and complementary integration of data-driven learning and geometrical modeling. We perform extensive experiments on the popular bench-marks, i.e., Human3.6M, 3DPW and SURREAL, where our PC-HMR frameworks achieve the SOTA results.
• Score: 47.601288796052714
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The end-to-end Human Mesh Recovery (HMR) approach has been successfully used for 3D body reconstruction. However, most HMR-based frameworks reconstruct human body by directly learning mesh parameters from images or videos, while lacking explicit guidance of 3D human pose in visual data. As a result, the generated mesh often exhibits incorrect pose for complex activities. To tackle this problem, we propose to exploit 3D pose to calibrate human mesh. Specifically, we develop two novel Pose Calibration frameworks, i.e., Serial PC-HMR and Parallel PC-HMR. By coupling advanced 3D pose estimators and HMR in a serial or parallel manner, these two frameworks can effectively correct human mesh with guidance of a concise pose calibration module. Furthermore, since the calibration module is designed via non-rigid pose transformation, our PC-HMR frameworks can flexibly tackle bone length variations to alleviate misplacement in the calibrated mesh. Finally, our frameworks are based on generic and complementary integration of data-driven learning and geometrical modeling. Via plug-and-play modules, they can be efficiently adapted for both image/video-based human mesh recovery. Additionally, they have no requirement of extra 3D pose annotations in the testing phase, which releases inference difficulties in practice. We perform extensive experiments on the popular bench-marks, i.e., Human3.6M, 3DPW and SURREAL, where our PC-HMR frameworks achieve the SOTA results.

Related papers

• Gamba: Marry Gaussian Splatting with Mamba for single view 3D reconstruction [153.52406455209538]
  Gamba is an end-to-end 3D reconstruction model from a single-view image. It completes reconstruction within 0.05 seconds on a single NVIDIA A100 GPU.
  arXiv  Detail & Related papers  (2024-03-27T17:40:14Z)
• Co-Evolution of Pose and Mesh for 3D Human Body Estimation from Video [23.93644678238666]
  We propose a Pose and Mesh Co-Evolution network (PMCE) to recover 3D human motion from a video. The proposed PMCE outperforms previous state-of-the-art methods in terms of both per-frame accuracy and temporal consistency.
  arXiv  Detail & Related papers  (2023-08-20T16:03:21Z)
• Sampling is Matter: Point-guided 3D Human Mesh Reconstruction [0.0]
  This paper presents a simple yet powerful method for 3D human mesh reconstruction from a single RGB image. Experimental results on benchmark datasets show that the proposed method efficiently improves the performance of 3D human mesh reconstruction.
  arXiv  Detail & Related papers  (2023-04-19T08:45:26Z)
• MPT: Mesh Pre-Training with Transformers for Human Pose and Mesh Reconstruction [56.80384196339199]
  Mesh Pre-Training (MPT) is a new pre-training framework that leverages 3D mesh data such as MoCap data for human pose and mesh reconstruction from a single image. MPT enables transformer models to have zero-shot capability of human mesh reconstruction from real images.
  arXiv  Detail & Related papers  (2022-11-24T00:02:13Z)
• PLIKS: A Pseudo-Linear Inverse Kinematic Solver for 3D Human Body Estimation [10.50175010474078]
  We introduce PLIKS for reconstruction of a 3D mesh of the human body from a single 2D image. PLIKS is built on a linearized formulation of the parametric SMPL model. We present evaluations which confirm that PLIKS achieves more accurate reconstruction with greater than 10% improvement.
  arXiv  Detail & Related papers  (2022-11-21T18:54:12Z)
• HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimation [39.67289969828706]
  We propose a novel hybrid inverse kinematics solution (HybrIK) to bridge the gap between body mesh estimation and 3D keypoint estimation. HybrIK directly transforms accurate 3D joints to relative body-part rotations for 3D body mesh reconstruction. We show that HybrIK preserves both the accuracy of 3D pose and the realistic body structure of the parametric human model.
  arXiv  Detail & Related papers  (2020-11-30T10:32:30Z)
• Synthetic Training for Monocular Human Mesh Recovery [100.38109761268639]
  This paper aims to estimate 3D mesh of multiple body parts with large-scale differences from a single RGB image. The main challenge is lacking training data that have complete 3D annotations of all body parts in 2D images. We propose a depth-to-scale (D2S) projection to incorporate the depth difference into the projection function to derive per-joint scale variants.
  arXiv  Detail & Related papers  (2020-10-27T03:31:35Z)
• Neural Descent for Visual 3D Human Pose and Shape [67.01050349629053]
  We present deep neural network methodology to reconstruct the 3d pose and shape of people, given an input RGB image. We rely on a recently introduced, expressivefull body statistical 3d human model, GHUM, trained end-to-end. Central to our methodology, is a learning to learn and optimize approach, referred to as HUmanNeural Descent (HUND), which avoids both second-order differentiation.
  arXiv  Detail & Related papers  (2020-08-16T13:38:41Z)
• SparseFusion: Dynamic Human Avatar Modeling from Sparse RGBD Images [49.52782544649703]
  We propose a novel approach to reconstruct 3D human body shapes based on a sparse set of RGBD frames. The main challenge is how to robustly fuse these sparse frames into a canonical 3D model. Our framework is flexible, with potential applications going beyond shape reconstruction.
  arXiv  Detail & Related papers  (2020-06-05T18:53:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.