HybrIK-X: Hybrid Analytical-Neural Inverse Kinematics for Whole-body Mesh Recovery

• URL: http://arxiv.org/abs/2304.05690v1
• Date: Wed, 12 Apr 2023 08:29:31 GMT
• Title: HybrIK-X: Hybrid Analytical-Neural Inverse Kinematics for Whole-body Mesh Recovery
• Authors: Jiefeng Li, Siyuan Bian, Chao Xu, Zhicun Chen, Lixin Yang, Cewu Lu
• Abstract summary: This paper presents a novel hybrid inverse kinematics solution, HybrIK, that integrates 3D keypoint estimation and body mesh recovery. HybrIK directly transforms accurate 3D joints to body-part rotations via twist-and-swing decomposition. We further develop a holistic framework, HybrIK-X, which enhances HybrIK with articulated hands and an expressive face.
• Score: 40.88628101598707
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recovering whole-body mesh by inferring the abstract pose and shape parameters from visual content can obtain 3D bodies with realistic structures. However, the inferring process is highly non-linear and suffers from image-mesh misalignment, resulting in inaccurate reconstruction. In contrast, 3D keypoint estimation methods utilize the volumetric representation to achieve pixel-level accuracy but may predict unrealistic body structures. To address these issues, this paper presents a novel hybrid inverse kinematics solution, HybrIK, that integrates the merits of 3D keypoint estimation and body mesh recovery in a unified framework. HybrIK directly transforms accurate 3D joints to body-part rotations via twist-and-swing decomposition. The swing rotations are analytically solved with 3D joints, while the twist rotations are derived from visual cues through neural networks. To capture comprehensive whole-body details, we further develop a holistic framework, HybrIK-X, which enhances HybrIK with articulated hands and an expressive face. HybrIK-X is fast and accurate by solving the whole-body pose with a one-stage model. Experiments demonstrate that HybrIK and HybrIK-X preserve both the accuracy of 3D joints and the realistic structure of the parametric human model, leading to pixel-aligned whole-body mesh recovery. The proposed method significantly surpasses the state-of-the-art methods on various benchmarks for body-only, hand-only, and whole-body scenarios. Code and results can be found at https://jeffli.site/HybrIK-X/

Related papers

• EndoSparse: Real-Time Sparse View Synthesis of Endoscopic Scenes using Gaussian Splatting [39.60431471170721]
  3D reconstruction of biological tissues from a collection of endoscopic images is a key to unlock various important downstream surgical applications with 3D capabilities. Existing methods employ various advanced neural rendering techniques for view synthesis, but they often struggle to recover accurate 3D representations when only sparse observations are available. We propose a framework leveraging the prior knowledge from multiple foundation models during the reconstruction process, dubbed as textitEndoSparse.
  arXiv  Detail & Related papers  (2024-07-01T07:24:09Z)
• NIKI: Neural Inverse Kinematics with Invertible Neural Networks for 3D Human Pose and Shape Estimation [53.25973084799954]
  We present NIKI (Neural Inverse Kinematics with Invertible Neural Network), which models bi-directional errors. NIKI can learn from both the forward and inverse processes with invertible networks.
  arXiv  Detail & Related papers  (2023-05-15T12:13:24Z)
• 3D Human Pose and Shape Estimation via HybrIK-Transformer [11.193504036335503]
  HybrIK relies on a combination of analytical inverse kinematics and deep learning to produce more accurate 3D pose estimation. We propose an enhancement of the 2D to 3D lifting module, replacing deconvolution with Transformer.
  arXiv  Detail & Related papers  (2023-02-09T17:08:43Z)
• PLIKS: A Pseudo-Linear Inverse Kinematic Solver for 3D Human Body Estimation [10.50175010474078]
  We introduce PLIKS for reconstruction of a 3D mesh of the human body from a single 2D image. PLIKS is built on a linearized formulation of the parametric SMPL model. We present evaluations which confirm that PLIKS achieves more accurate reconstruction with greater than 10% improvement.
  arXiv  Detail & Related papers  (2022-11-21T18:54:12Z)
• Detailed Avatar Recovery from Single Image [50.82102098057822]
  This paper presents a novel framework to recover emphdetailed avatar from a single image. We use the deep neural networks to refine the 3D shape in a Hierarchical Mesh Deformation framework. Our method can restore detailed human body shapes with complete textures beyond skinned models.
  arXiv  Detail & Related papers  (2021-08-06T03:51:26Z)
• KAMA: 3D Keypoint Aware Body Mesh Articulation [79.04090630502782]
  We propose an analytical solution to articulate a parametric body model, SMPL, via a set of straightforward geometric transformations. Our approach offers significantly better alignment to image content when compared to state-of-the-art approaches. Results on the challenging 3DPW and Human3.6M demonstrate that our approach yields state-of-the-art body mesh fittings.
  arXiv  Detail & Related papers  (2021-04-27T23:01:03Z)
• PC-HMR: Pose Calibration for 3D Human Mesh Recovery from 2D Images/Videos [47.601288796052714]
  We develop two novel Pose frameworks, i.e., Serial PC-HMR and Parallel PC-HMR. Our frameworks are based on generic and complementary integration of data-driven learning and geometrical modeling. We perform extensive experiments on the popular bench-marks, i.e., Human3.6M, 3DPW and SURREAL, where our PC-HMR frameworks achieve the SOTA results.
  arXiv  Detail & Related papers  (2021-03-16T12:12:45Z)
• HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimation [39.67289969828706]
  We propose a novel hybrid inverse kinematics solution (HybrIK) to bridge the gap between body mesh estimation and 3D keypoint estimation. HybrIK directly transforms accurate 3D joints to relative body-part rotations for 3D body mesh reconstruction. We show that HybrIK preserves both the accuracy of 3D pose and the realistic body structure of the parametric human model.
  arXiv  Detail & Related papers  (2020-11-30T10:32:30Z)
• SparseFusion: Dynamic Human Avatar Modeling from Sparse RGBD Images [49.52782544649703]
  We propose a novel approach to reconstruct 3D human body shapes based on a sparse set of RGBD frames. The main challenge is how to robustly fuse these sparse frames into a canonical 3D model. Our framework is flexible, with potential applications going beyond shape reconstruction.
  arXiv  Detail & Related papers  (2020-06-05T18:53:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.