Hybrid 3D Human Pose Estimation with Monocular Video and Sparse IMUs

• URL: http://arxiv.org/abs/2404.17837v1
• Date: Sat, 27 Apr 2024 09:02:42 GMT
• Title: Hybrid 3D Human Pose Estimation with Monocular Video and Sparse IMUs
• Authors: Yiming Bao, Xu Zhao, Dahong Qian,
• Abstract summary: Temporal 3D human pose estimation from monocular videos is a challenging task in human-centered computer vision. Inertial sensor has been introduced to provide complementary source of information. It remains challenging to integrate heterogeneous sensor data for producing physically rational 3D human poses.
• Score: 15.017274891943162
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Temporal 3D human pose estimation from monocular videos is a challenging task in human-centered computer vision due to the depth ambiguity of 2D-to-3D lifting. To improve accuracy and address occlusion issues, inertial sensor has been introduced to provide complementary source of information. However, it remains challenging to integrate heterogeneous sensor data for producing physically rational 3D human poses. In this paper, we propose a novel framework, Real-time Optimization and Fusion (RTOF), to address this issue. We first incorporate sparse inertial orientations into a parametric human skeleton to refine 3D poses in kinematics. The poses are then optimized by energy functions built on both visual and inertial observations to reduce the temporal jitters. Our framework outputs smooth and biomechanically plausible human motion. Comprehensive experiments with ablation studies demonstrate its rationality and efficiency. On Total Capture dataset, the pose estimation error is significantly decreased compared to the baseline method.

Related papers

• Enhancing 3D Human Pose Estimation Amidst Severe Occlusion with Dual Transformer Fusion [13.938406073551844]
  This paper introduces a Dual Transformer Fusion (DTF) algorithm, a novel approach to obtain a holistic 3D pose estimation. To enable precise 3D Human Pose Estimation, our approach leverages the innovative DTF architecture, which first generates a pair of intermediate views. Our approach outperforms existing state-of-the-art methods on both datasets, yielding substantial improvements.
  arXiv  Detail & Related papers  (2024-10-06T18:15:27Z)
• Occlusion-Aware 3D Motion Interpretation for Abnormal Behavior Detection [10.782354892545651]
  We present OAD2D, which discriminates against motion abnormalities based on reconstructing 3D coordinates of mesh vertices and human joints from monocular videos. We reformulate the abnormal posture estimation by coupling it with Motion to Text (M2T) model in which, the VQVAE is employed to quantize motion features. Our approach demonstrates the robustness of abnormal behavior detection against severe and self-occlusions, as it reconstructs human motion trajectories in global coordinates.
  arXiv  Detail & Related papers  (2024-07-23T18:41:16Z)
• Zolly: Zoom Focal Length Correctly for Perspective-Distorted Human Mesh Reconstruction [66.10717041384625]
  Zolly is the first 3DHMR method focusing on perspective-distorted images. We propose a new camera model and a novel 2D representation, termed distortion image, which describes the 2D dense distortion scale of the human body. We extend two real-world datasets tailored for this task, all containing perspective-distorted human images.
  arXiv  Detail & Related papers  (2023-03-24T04:22:41Z)
• DiffuPose: Monocular 3D Human Pose Estimation via Denoising Diffusion Probabilistic Model [25.223801390996435]
  This paper focuses on reconstructing a 3D pose from a single 2D keypoint detection. We build a novel diffusion-based framework to effectively sample diverse 3D poses from an off-the-shelf 2D detector. We evaluate our method on the widely adopted Human3.6M and HumanEva-I datasets.
  arXiv  Detail & Related papers  (2022-12-06T07:22:20Z)
• Occluded Human Body Capture with Self-Supervised Spatial-Temporal Motion Prior [7.157324258813676]
  We build the first 3D occluded motion dataset(OcMotion), which can be used for both training and testing. A spatial-temporal layer is then designed to learn joint-level correlations. Experimental results show that our method can generate accurate and coherent human motions from occluded videos with good generalization ability and runtime efficiency.
  arXiv  Detail & Related papers  (2022-07-12T08:15:11Z)
• Homography Loss for Monocular 3D Object Detection [54.04870007473932]
  A differentiable loss function, termed as Homography Loss, is proposed to achieve the goal, which exploits both 2D and 3D information. Our method yields the best performance compared with the other state-of-the-arts by a large margin on KITTI 3D datasets.
  arXiv  Detail & Related papers  (2022-04-02T03:48:03Z)
• Recovering 3D Human Mesh from Monocular Images: A Survey [49.00136388529404]
  Estimating human pose and shape from monocular images is a long-standing problem in computer vision. This survey focuses on the task of monocular 3D human mesh recovery.
  arXiv  Detail & Related papers  (2022-03-03T18:56:08Z)
• 3D Skeleton-based Human Motion Prediction with Manifold-Aware GAN [3.1313293632309827]
  We propose a novel solution for 3D skeleton-based human motion prediction. We build a manifold-aware Wasserstein generative adversarial model that captures the temporal and spatial dependencies of human motion. Experiments have been conducted on CMU MoCap and Human 3.6M datasets.
  arXiv  Detail & Related papers  (2022-03-01T20:49:13Z)
• Neural Monocular 3D Human Motion Capture with Physical Awareness [76.55971509794598]
  We present a new trainable system for physically plausible markerless 3D human motion capture. Unlike most neural methods for human motion capture, our approach is aware of physical and environmental constraints. It produces smooth and physically principled 3D motions in an interactive frame rate in a wide variety of challenging scenes.
  arXiv  Detail & Related papers  (2021-05-03T17:57:07Z)
• Synthetic Training for Monocular Human Mesh Recovery [100.38109761268639]
  This paper aims to estimate 3D mesh of multiple body parts with large-scale differences from a single RGB image. The main challenge is lacking training data that have complete 3D annotations of all body parts in 2D images. We propose a depth-to-scale (D2S) projection to incorporate the depth difference into the projection function to derive per-joint scale variants.
  arXiv  Detail & Related papers  (2020-10-27T03:31:35Z)
• Anatomy-aware 3D Human Pose Estimation with Bone-based Pose Decomposition [92.99291528676021]
  Instead of directly regressing the 3D joint locations, we decompose the task into bone direction prediction and bone length prediction. Our motivation is the fact that the bone lengths of a human skeleton remain consistent across time. Our full model outperforms the previous best results on Human3.6M and MPI-INF-3DHP datasets.
  arXiv  Detail & Related papers  (2020-02-24T15:49:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.