CameraHMR: Aligning People with Perspective

• URL: http://arxiv.org/abs/2411.08128v1
• Date: Tue, 12 Nov 2024 19:12:12 GMT
• Title: CameraHMR: Aligning People with Perspective
• Authors: Priyanka Patel, Michael J. Black,
• Abstract summary: We address the challenge of accurate 3D human pose and shape estimation from monocular images. Existing training datasets containing real images with pseudo ground truth (pGT) use SMPLify to fit SMPL to sparse 2D joint locations. We make two contributions that improve pGT accuracy.
• Score: 54.05758012879385
• License:
• Abstract: We address the challenge of accurate 3D human pose and shape estimation from monocular images. The key to accuracy and robustness lies in high-quality training data. Existing training datasets containing real images with pseudo ground truth (pGT) use SMPLify to fit SMPL to sparse 2D joint locations, assuming a simplified camera with default intrinsics. We make two contributions that improve pGT accuracy. First, to estimate camera intrinsics, we develop a field-of-view prediction model (HumanFoV) trained on a dataset of images containing people. We use the estimated intrinsics to enhance the 4D-Humans dataset by incorporating a full perspective camera model during SMPLify fitting. Second, 2D joints provide limited constraints on 3D body shape, resulting in average-looking bodies. To address this, we use the BEDLAM dataset to train a dense surface keypoint detector. We apply this detector to the 4D-Humans dataset and modify SMPLify to fit the detected keypoints, resulting in significantly more realistic body shapes. Finally, we upgrade the HMR2.0 architecture to include the estimated camera parameters. We iterate model training and SMPLify fitting initialized with the previously trained model. This leads to more accurate pGT and a new model, CameraHMR, with state-of-the-art accuracy. Code and pGT are available for research purposes.

Related papers

• Neural Localizer Fields for Continuous 3D Human Pose and Shape Estimation [32.30055363306321]
  We propose a paradigm for seamlessly unifying different human pose and shape-related tasks and datasets. Our formulation is centered on the ability - both at training and test time - to query any arbitrary point of the human volume. We can naturally exploit differently annotated data sources including mesh, 2D/3D skeleton and dense pose, without having to convert between them.
  arXiv  Detail & Related papers  (2024-07-10T10:44:18Z)
• Zolly: Zoom Focal Length Correctly for Perspective-Distorted Human Mesh Reconstruction [66.10717041384625]
  Zolly is the first 3DHMR method focusing on perspective-distorted images. We propose a new camera model and a novel 2D representation, termed distortion image, which describes the 2D dense distortion scale of the human body. We extend two real-world datasets tailored for this task, all containing perspective-distorted human images.
  arXiv  Detail & Related papers  (2023-03-24T04:22:41Z)
• Adversarial Parametric Pose Prior [106.12437086990853]
  We learn a prior that restricts the SMPL parameters to values that produce realistic poses via adversarial training. We show that our learned prior covers the diversity of the real-data distribution, facilitates optimization for 3D reconstruction from 2D keypoints, and yields better pose estimates when used for regression from images.
  arXiv  Detail & Related papers  (2021-12-08T10:05:32Z)
• Shape-aware Multi-Person Pose Estimation from Multi-View Images [47.13919147134315]
  Our proposed coarse-to-fine pipeline first aggregates noisy 2D observations from multiple camera views into 3D space. The final pose estimates are attained from a novel optimization scheme which links high-confidence multi-view 2D observations and 3D joint candidates.
  arXiv  Detail & Related papers  (2021-10-05T20:04:21Z)
• SPEC: Seeing People in the Wild with an Estimated Camera [64.85791231401684]
  We introduce SPEC, the first in-the-wild 3D HPS method that estimates the perspective camera from a single image. We train a neural network to estimate the field of view, camera pitch, and roll an input image. We then train a novel network that rolls the camera calibration to the image features and uses these together to regress 3D body shape and pose.
  arXiv  Detail & Related papers  (2021-10-01T19:05:18Z)
• Uncertainty-Aware Camera Pose Estimation from Points and Lines [101.03675842534415]
  Perspective-n-Point-and-Line (Pn$PL) aims at fast, accurate and robust camera localizations with respect to a 3D model from 2D-3D feature coordinates.
  arXiv  Detail & Related papers  (2021-07-08T15:19:36Z)
• Synthetic Training for Monocular Human Mesh Recovery [100.38109761268639]
  This paper aims to estimate 3D mesh of multiple body parts with large-scale differences from a single RGB image. The main challenge is lacking training data that have complete 3D annotations of all body parts in 2D images. We propose a depth-to-scale (D2S) projection to incorporate the depth difference into the projection function to derive per-joint scale variants.
  arXiv  Detail & Related papers  (2020-10-27T03:31:35Z)
• Beyond Weak Perspective for Monocular 3D Human Pose Estimation [6.883305568568084]
  We consider the task of 3D joints location and orientation prediction from a monocular video. We first infer 2D joints locations with an off-the-shelf pose estimation algorithm. We then adhere to the SMPLify algorithm which receives those initial parameters.
  arXiv  Detail & Related papers  (2020-09-14T16:23:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.