Lifting 2D Human Pose to 3D with Domain Adapted 3D Body Concept

• URL: http://arxiv.org/abs/2111.11969v1
• Date: Tue, 23 Nov 2021 16:02:12 GMT
• Title: Lifting 2D Human Pose to 3D with Domain Adapted 3D Body Concept
• Authors: Qiang Nie, Ziwei Liu, Yunhui Liu
• Abstract summary: Existing 3D pose estimation suffers from 1) the inherent ambiguity between the 2D and 3D data, and 2) the lack of well labeled 2D-3D pose pairs in the wild. We propose a new framework that leverages the labeled 3D human poses to learn a 3D concept of the human body to reduce the ambiguity. By adapting the two domains, the body knowledge learned from 3D poses is applied to 2D poses and guides the 2D pose encoder to generate informative 3D "imagination" as embedding in pose lifting.
• Score: 49.49032810966848
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Lifting the 2D human pose to the 3D pose is an important yet challenging task. Existing 3D pose estimation suffers from 1) the inherent ambiguity between the 2D and 3D data, and 2) the lack of well labeled 2D-3D pose pairs in the wild. Human beings are able to imagine the human 3D pose from a 2D image or a set of 2D body key-points with the least ambiguity, which should be attributed to the prior knowledge of the human body that we have acquired in our mind. Inspired by this, we propose a new framework that leverages the labeled 3D human poses to learn a 3D concept of the human body to reduce the ambiguity. To have consensus on the body concept from 2D pose, our key insight is to treat the 2D human pose and the 3D human pose as two different domains. By adapting the two domains, the body knowledge learned from 3D poses is applied to 2D poses and guides the 2D pose encoder to generate informative 3D "imagination" as embedding in pose lifting. Benefiting from the domain adaptation perspective, the proposed framework unifies the supervised and semi-supervised 3D pose estimation in a principled framework. Extensive experiments demonstrate that the proposed approach can achieve state-of-the-art performance on standard benchmarks. More importantly, it is validated that the explicitly learned 3D body concept effectively alleviates the 2D-3D ambiguity in 2D pose lifting, improves the generalization, and enables the network to exploit the abundant unlabeled 2D data.

Related papers

• MPL: Lifting 3D Human Pose from Multi-view 2D Poses [75.26416079541723]
  We propose combining 2D pose estimation, for which large and rich training datasets exist, and 2D-to-3D pose lifting, using a transformer-based network. Our experiments demonstrate decreases up to 45% in MPJPE errors compared to the 3D pose obtained by triangulating the 2D poses.
  arXiv  Detail & Related papers  (2024-08-20T12:55:14Z)
• Cloth2Body: Generating 3D Human Body Mesh from 2D Clothing [54.29207348918216]
  Cloth2Body needs to address new and emerging challenges raised by the partial observation of the input and the high diversity of the output. We propose an end-to-end framework that can accurately estimate 3D body mesh parameterized by pose and shape from a 2D clothing image. As shown by experimental results, the proposed framework achieves state-of-the-art performance and can effectively recover natural and diverse 3D body meshes from 2D images.
  arXiv  Detail & Related papers  (2023-09-28T06:18:38Z)
• MPM: A Unified 2D-3D Human Pose Representation via Masked Pose Modeling [59.74064212110042]
  mpmcan handle multiple tasks including 3D human pose estimation, 3D pose estimation from cluded 2D pose, and 3D pose completion in a textocbfsingle framework. We conduct extensive experiments and ablation studies on several widely used human pose datasets and achieve state-of-the-art performance on MPI-INF-3DHP.
  arXiv  Detail & Related papers  (2023-06-29T10:30:00Z)
• Weakly-supervised Pre-training for 3D Human Pose Estimation via Perspective Knowledge [36.65402869749077]
  We propose a novel method to extract weak 3D information directly from 2D images without 3D pose supervision. We propose a weakly-supervised pre-training (WSP) strategy to distinguish the depth relationship between two points in an image. WSP achieves state-of-the-art results on two widely-used benchmarks.
  arXiv  Detail & Related papers  (2022-11-22T03:35:15Z)
• SPGNet: Spatial Projection Guided 3D Human Pose Estimation in Low Dimensional Space [14.81199315166042]
  We propose a method for 3D human pose estimation that mixes multi-dimensional re-projection into supervised learning. Based on the estimation results for the dataset Human3.6M, our approach outperforms many state-of-the-art methods both qualitatively and quantitatively.
  arXiv  Detail & Related papers  (2022-06-04T00:51:00Z)
• 3DCrowdNet: 2D Human Pose-Guided3D Crowd Human Pose and Shape Estimation in the Wild [61.92656990496212]
  3DCrowdNet is a 2D human pose-guided 3D crowd pose and shape estimation system for in-the-wild scenes. We show that our 3DCrowdNet outperforms previous methods on in-the-wild crowd scenes.
  arXiv  Detail & Related papers  (2021-04-15T08:21:28Z)
• Deep Monocular 3D Human Pose Estimation via Cascaded Dimension-Lifting [10.336146336350811]
  3D pose estimation from a single image is a challenging problem due to depth ambiguity. One type of the previous methods lifts 2D joints, obtained by resorting to external 2D pose detectors, to the 3D space. We propose a novel end-to-end framework that exploits the contextual information but also produces the output directly in the 3D space.
  arXiv  Detail & Related papers  (2021-04-08T05:44:02Z)
• Unsupervised 3D Human Pose Representation with Viewpoint and Pose Disentanglement [63.853412753242615]
  Learning a good 3D human pose representation is important for human pose related tasks. We propose a novel Siamese denoising autoencoder to learn a 3D pose representation. Our approach achieves state-of-the-art performance on two inherently different tasks.
  arXiv  Detail & Related papers  (2020-07-14T14:25:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.