Monocular Human Pose and Shape Reconstruction using Part Differentiable Rendering

• URL: http://arxiv.org/abs/2003.10873v2
• Date: Fri, 29 Jan 2021 08:57:26 GMT
• Title: Monocular Human Pose and Shape Reconstruction using Part Differentiable Rendering
• Authors: Min Wang, Feng Qiu, Wentao Liu, Chen Qian, Xiaowei Zhou, Lizhuang Ma
• Abstract summary: Recent works succeed in regression-based methods which estimate parametric models directly through a deep neural network supervised by 3D ground truth. In this paper, we introduce body segmentation as critical supervision. To improve the reconstruction with part segmentation, we propose a part-level differentiable part that enables part-based models to be supervised by part segmentation.
• Score: 53.16864661460889
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Superior human pose and shape reconstruction from monocular images depends on removing the ambiguities caused by occlusions and shape variance. Recent works succeed in regression-based methods which estimate parametric models directly through a deep neural network supervised by 3D ground truth. However, 3D ground truth is neither in abundance nor can efficiently be obtained. In this paper, we introduce body part segmentation as critical supervision. Part segmentation not only indicates the shape of each body part but helps to infer the occlusions among parts as well. To improve the reconstruction with part segmentation, we propose a part-level differentiable renderer that enables part-based models to be supervised by part segmentation in neural networks or optimization loops. We also introduce a general parametric model engaged in the rendering pipeline as an intermediate representation between skeletons and detailed shapes, which consists of primitive geometries for better interpretability. The proposed approach combines parameter regression, body model optimization, and detailed model registration altogether. Experimental results demonstrate that the proposed method achieves balanced evaluation on pose and shape, and outperforms the state-of-the-art approaches on Human3.6M, UP-3D and LSP datasets.

Related papers

• DeFormer: Integrating Transformers with Deformable Models for 3D Shape Abstraction from a Single Image [31.154786931081087]
  We propose a novel bi-channel Transformer architecture, integrated with parameterized deformable models, to simultaneously estimate the global and local deformations of primitives. DeFormer achieves better reconstruction accuracy over the state-of-the-art, and visualizes with consistent semantic correspondences for improved interpretability.
  arXiv  Detail & Related papers  (2023-09-22T02:46:43Z)
• Unsupervised 3D Pose Estimation with Non-Rigid Structure-from-Motion Modeling [83.76377808476039]
  We propose a new modeling method for human pose deformations and design an accompanying diffusion-based motion prior. Inspired by the field of non-rigid structure-from-motion, we divide the task of reconstructing 3D human skeletons in motion into the estimation of a 3D reference skeleton. A mixed spatial-temporal NRSfMformer is used to simultaneously estimate the 3D reference skeleton and the skeleton deformation of each frame from 2D observations sequence.
  arXiv  Detail & Related papers  (2023-08-18T16:41:57Z)
• ANISE: Assembly-based Neural Implicit Surface rEconstruction [12.745433575962842]
  We present ANISE, a method that reconstructs a 3Dshape from partial observations (images or sparse point clouds) The shape is formulated as an assembly of neural implicit functions, each representing a different part instance. We demonstrate that, when performing reconstruction by decoding part representations into implicit functions, our method achieves state-of-the-art part-aware reconstruction results from both images and sparse point clouds.
  arXiv  Detail & Related papers  (2022-05-27T00:01:40Z)
• The Best of Both Worlds: Combining Model-based and Nonparametric Approaches for 3D Human Body Estimation [20.797162096899154]
  We propose a framework for estimating model parameters from global image features. A dense map prediction module explicitly establishes the dense UV correspondence between the image evidence and each part of the body model. In inverse kinematics module refines the key point prediction and generates a posed template mesh. A UV inpainting module relies on the corresponding feature, prediction and the posed template, and completes the predictions of occluded body shape.
  arXiv  Detail & Related papers  (2022-05-01T16:39:09Z)
• LatentHuman: Shape-and-Pose Disentangled Latent Representation for Human Bodies [78.17425779503047]
  We propose a novel neural implicit representation for the human body. It is fully differentiable and optimizable with disentangled shape and pose latent spaces. Our model can be trained and fine-tuned directly on non-watertight raw data with well-designed losses.
  arXiv  Detail & Related papers  (2021-11-30T04:10:57Z)
• Neural Descent for Visual 3D Human Pose and Shape [67.01050349629053]
  We present deep neural network methodology to reconstruct the 3d pose and shape of people, given an input RGB image. We rely on a recently introduced, expressivefull body statistical 3d human model, GHUM, trained end-to-end. Central to our methodology, is a learning to learn and optimize approach, referred to as HUmanNeural Descent (HUND), which avoids both second-order differentiation.
  arXiv  Detail & Related papers  (2020-08-16T13:38:41Z)
• Learning Nonparametric Human Mesh Reconstruction from a Single Image without Ground Truth Meshes [56.27436157101251]
  We propose a novel approach to learn human mesh reconstruction without any ground truth meshes. This is made possible by introducing two new terms into the loss function of a graph convolutional neural network (Graph CNN)
  arXiv  Detail & Related papers  (2020-02-28T20:30:07Z)
• Learning 3D Human Shape and Pose from Dense Body Parts [117.46290013548533]
  We propose a Decompose-and-aggregate Network (DaNet) to learn 3D human shape and pose from dense correspondences of body parts. Messages from local streams are aggregated to enhance the robust prediction of the rotation-based poses. Our method is validated on both indoor and real-world datasets including Human3.6M, UP3D, COCO, and 3DPW.
  arXiv  Detail & Related papers  (2019-12-31T15:09:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.