SkelFormer: Markerless 3D Pose and Shape Estimation using Skeletal Transformers

• URL: http://arxiv.org/abs/2404.12625v1
• Date: Fri, 19 Apr 2024 04:51:18 GMT
• Title: SkelFormer: Markerless 3D Pose and Shape Estimation using Skeletal Transformers
• Authors: Vandad Davoodnia, Saeed Ghorbani, Alexandre Messier, Ali Etemad,
• Abstract summary: We introduce SkelFormer, a novel markerless motion capture pipeline for multi-view human pose and shape estimation. Our method first uses off-the-shelf 2D keypoint estimators, pre-trained on large-scale in-the-wild data, to obtain 3D joint positions. Next, we design a regression-based inverse-kinematic skeletal transformer that maps the joint positions to pose and shape representations from heavily noisy observations.
• Score: 57.46911575980854
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce SkelFormer, a novel markerless motion capture pipeline for multi-view human pose and shape estimation. Our method first uses off-the-shelf 2D keypoint estimators, pre-trained on large-scale in-the-wild data, to obtain 3D joint positions. Next, we design a regression-based inverse-kinematic skeletal transformer that maps the joint positions to pose and shape representations from heavily noisy observations. This module integrates prior knowledge about pose space and infers the full pose state at runtime. Separating the 3D keypoint detection and inverse-kinematic problems, along with the expressive representations learned by our skeletal transformer, enhance the generalization of our method to unseen noisy data. We evaluate our method on three public datasets in both in-distribution and out-of-distribution settings using three datasets, and observe strong performance with respect to prior works. Moreover, ablation experiments demonstrate the impact of each of the modules of our architecture. Finally, we study the performance of our method in dealing with noise and heavy occlusions and find considerable robustness with respect to other solutions.

Related papers

• PF3plat: Pose-Free Feed-Forward 3D Gaussian Splatting [54.7468067660037]
  PF3plat sets a new state-of-the-art across all benchmarks, supported by comprehensive ablation studies validating our design choices. Our framework capitalizes on fast speed, scalability, and high-quality 3D reconstruction and view synthesis capabilities of 3DGS.
  arXiv  Detail & Related papers  (2024-10-29T15:28:15Z)
• UPose3D: Uncertainty-Aware 3D Human Pose Estimation with Cross-View and Temporal Cues [55.69339788566899]
  UPose3D is a novel approach for multi-view 3D human pose estimation. It improves robustness and flexibility without requiring direct 3D annotations.
  arXiv  Detail & Related papers  (2024-04-23T00:18:00Z)
• Multiple View Geometry Transformers for 3D Human Pose Estimation [35.26756920323391]
  We aim to improve the 3D reasoning ability of Transformers in multi-view 3D human pose estimation. We propose a novel hybrid model, MVGFormer, which has a series of geometric and appearance modules organized in an iterative manner.
  arXiv  Detail & Related papers  (2023-11-18T06:32:40Z)
• Unsupervised 3D Pose Estimation with Non-Rigid Structure-from-Motion Modeling [83.76377808476039]
  We propose a new modeling method for human pose deformations and design an accompanying diffusion-based motion prior. Inspired by the field of non-rigid structure-from-motion, we divide the task of reconstructing 3D human skeletons in motion into the estimation of a 3D reference skeleton. A mixed spatial-temporal NRSfMformer is used to simultaneously estimate the 3D reference skeleton and the skeleton deformation of each frame from 2D observations sequence.
  arXiv  Detail & Related papers  (2023-08-18T16:41:57Z)
• Weakly-supervised 3D Pose Transfer with Keypoints [57.66991032263699]
  Main challenges of 3D pose transfer are: 1) Lack of paired training data with different characters performing the same pose; 2) Disentangling pose and shape information from the target mesh; 3) Difficulty in applying to meshes with different topologies. We propose a novel weakly-supervised keypoint-based framework to overcome these difficulties.
  arXiv  Detail & Related papers  (2023-07-25T12:40:24Z)
• Multi-Person 3D Pose and Shape Estimation via Inverse Kinematics and Refinement [5.655207244072081]
  Estimating 3D poses and shapes in the form of meshes from monocular RGB images is challenging. We propose a coarse-to-fine pipeline that benefits from 1) inverse kinematics from the occlusion-robust 3D skeleton estimation. We demonstrate the effectiveness of our method, outperforming state-of-the-arts on 3DPW, MuPoTS and AGORA datasets.
  arXiv  Detail & Related papers  (2022-10-24T18:29:06Z)
• 3D Human Pose Estimation with Spatial and Temporal Transformers [59.433208652418976]
  We present PoseFormer, a purely transformer-based approach for 3D human pose estimation in videos. Inspired by recent developments in vision transformers, we design a spatial-temporal transformer structure. We quantitatively and qualitatively evaluate our method on two popular and standard benchmark datasets.
  arXiv  Detail & Related papers  (2021-03-18T18:14:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.