3D Human Pose and Shape Regression with Pyramidal Mesh Alignment Feedback Loop

• URL: http://arxiv.org/abs/2103.16507v2
• Date: Thu, 1 Apr 2021 11:46:56 GMT
• Title: 3D Human Pose and Shape Regression with Pyramidal Mesh Alignment Feedback Loop
• Authors: Hongwen Zhang, Yating Tian, Xinchi Zhou, Wanli Ouyang, Yebin Liu, Limin Wang, Zhenan Sun
• Abstract summary: Regression-based methods have recently shown promising results in reconstructing human meshes from monocular images. Minor deviation in parameters may lead to noticeable misalignment between the estimated meshes and image evidences. We propose a Pyramidal Mesh Alignment Feedback (PyMAF) loop to leverage a feature pyramid and rectify the predicted parameters.
• Score: 128.07841893637337
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Regression-based methods have recently shown promising results in reconstructing human meshes from monocular images. By directly mapping from raw pixels to model parameters, these methods can produce parametric models in a feed-forward manner via neural networks. However, minor deviation in parameters may lead to noticeable misalignment between the estimated meshes and image evidences. To address this issue, we propose a Pyramidal Mesh Alignment Feedback (PyMAF) loop to leverage a feature pyramid and rectify the predicted parameters explicitly based on the mesh-image alignment status in our deep regressor. In PyMAF, given the currently predicted parameters, mesh-aligned evidences will be extracted from finer-resolution features accordingly and fed back for parameter rectification. To reduce noise and enhance the reliability of these evidences, an auxiliary pixel-wise supervision is imposed on the feature encoder, which provides mesh-image correspondence guidance for our network to preserve the most related information in spatial features. The efficacy of our approach is validated on several benchmarks, including Human3.6M, 3DPW, LSP, and COCO, where experimental results show that our approach consistently improves the mesh-image alignment of the reconstruction. Our code is publicly available at https://hongwenzhang.github.io/pymaf .

Related papers

• PF3plat: Pose-Free Feed-Forward 3D Gaussian Splatting [54.7468067660037]
  PF3plat sets a new state-of-the-art across all benchmarks, supported by comprehensive ablation studies validating our design choices. Our framework capitalizes on fast speed, scalability, and high-quality 3D reconstruction and view synthesis capabilities of 3DGS.
  arXiv  Detail & Related papers  (2024-10-29T15:28:15Z)
• Delving Deep into Pixel Alignment Feature for Accurate Multi-view Human Mesh Recovery [37.57922952189394]
  We present Pixel-aligned Feedback Fusion (PaFF) for accurate yet efficient human mesh recovery from multi-view images. PaFF is an iterative regression framework that performs feature extraction and fusion alternately. The efficacy of our method is validated in the Human3.6M dataset via comprehensive ablation experiments.
  arXiv  Detail & Related papers  (2023-01-15T05:31:52Z)
• Learnable human mesh triangulation for 3D human pose and shape estimation [6.699132260402631]
  The accuracy of joint rotation and shape estimation has received relatively little attention in the skinned multi-person linear model (SMPL)-based human mesh reconstruction from multi-view images. We propose a two-stage method to resolve the ambiguity of joint rotation and shape reconstruction and the difficulty of network learning. The proposed method significantly outperforms the previous works in terms of joint rotation and shape estimation, and achieves competitive performance in terms of joint location estimation.
  arXiv  Detail & Related papers  (2022-08-24T01:11:57Z)
• PyMAF-X: Towards Well-aligned Full-body Model Regression from Monocular Images [60.33197938330409]
  PyMAF-X is a regression-based approach to recovering parametric full-body models from monocular images. PyMAF and PyMAF-X effectively improve the mesh-image alignment and achieve new state-of-the-art results.
  arXiv  Detail & Related papers  (2022-07-13T17:58:33Z)
• A Model for Multi-View Residual Covariances based on Perspective Deformation [88.21738020902411]
  We derive a model for the covariance of the visual residuals in multi-view SfM, odometry and SLAM setups. We validate our model with synthetic and real data and integrate it into photometric and feature-based Bundle Adjustment.
  arXiv  Detail & Related papers  (2022-02-01T21:21:56Z)
• Adversarial Parametric Pose Prior [106.12437086990853]
  We learn a prior that restricts the SMPL parameters to values that produce realistic poses via adversarial training. We show that our learned prior covers the diversity of the real-data distribution, facilitates optimization for 3D reconstruction from 2D keypoints, and yields better pose estimates when used for regression from images.
  arXiv  Detail & Related papers  (2021-12-08T10:05:32Z)
• Ladybird: Quasi-Monte Carlo Sampling for Deep Implicit Field Based 3D Reconstruction with Symmetry [12.511526058118143]
  We propose a sampling scheme that theoretically encourages generalization and results in fast convergence for SGD-based optimization algorithms. Based on the reflective symmetry of an object, we propose a feature fusion method that alleviates issues due to self-occlusions. Our proposed system Ladybird is able to create high quality 3D object reconstructions from a single input image.
  arXiv  Detail & Related papers  (2020-07-27T09:17:00Z)
• Learning Nonparametric Human Mesh Reconstruction from a Single Image without Ground Truth Meshes [56.27436157101251]
  We propose a novel approach to learn human mesh reconstruction without any ground truth meshes. This is made possible by introducing two new terms into the loss function of a graph convolutional neural network (Graph CNN)
  arXiv  Detail & Related papers  (2020-02-28T20:30:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.