Overcoming the Trade-off Between Accuracy and Plausibility in 3D Hand Shape Reconstruction

• URL: http://arxiv.org/abs/2305.00646v1
• Date: Mon, 1 May 2023 03:38:01 GMT
• Title: Overcoming the Trade-off Between Accuracy and Plausibility in 3D Hand Shape Reconstruction
• Authors: Ziwei Yu, Chen Li, Linlin Yang, Xiaoxu Zheng, Michael Bi Mi, Gim Hee Lee, Angela Yao
• Abstract summary: Direct mesh fitting for 3D hand shape reconstruction is highly accurate. However, the reconstructed meshes are prone to artifacts and do not appear as plausible hand shapes. We introduce a novel weakly-supervised hand shape estimation framework that integrates non-parametric mesh fitting with MANO model in an end-to-end fashion.
• Score: 62.96478903239799
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Direct mesh fitting for 3D hand shape reconstruction is highly accurate. However, the reconstructed meshes are prone to artifacts and do not appear as plausible hand shapes. Conversely, parametric models like MANO ensure plausible hand shapes but are not as accurate as the non-parametric methods. In this work, we introduce a novel weakly-supervised hand shape estimation framework that integrates non-parametric mesh fitting with MANO model in an end-to-end fashion. Our joint model overcomes the tradeoff in accuracy and plausibility to yield well-aligned and high-quality 3D meshes, especially in challenging two-hand and hand-object interaction scenarios.

Related papers

• 3D Hand Reconstruction via Aggregating Intra and Inter Graphs Guided by Prior Knowledge for Hand-Object Interaction Scenario [8.364378460776832]
  We propose a 3D hand reconstruction network combining the benefits of model-based and model-free approaches to balance accuracy and physical plausibility for hand-object interaction scenario. Firstly, we present a novel MANO pose parameters regression module from 2D joints directly, which avoids the process of highly nonlinear mapping from abstract image feature.
  arXiv  Detail & Related papers  (2024-03-04T05:11:26Z)
• Monocular 3D Hand Mesh Recovery via Dual Noise Estimation [47.82179706128616]
  We introduce a dual noise estimation method to generate meshes that are aligned with the image well. Our method achieves state-of-the-art performance on a large-scale Interhand2.6M dataset.
  arXiv  Detail & Related papers  (2023-12-26T07:21:01Z)
• Decaf: Monocular Deformation Capture for Face and Hand Interactions [77.75726740605748]
  This paper introduces the first method that allows tracking human hands interacting with human faces in 3D from single monocular RGB videos. We model hands as articulated objects inducing non-rigid face deformations during an active interaction. Our method relies on a new hand-face motion and interaction capture dataset with realistic face deformations acquired with a markerless multi-view camera system.
  arXiv  Detail & Related papers  (2023-09-28T17:59:51Z)
• PHRIT: Parametric Hand Representation with Implicit Template [24.699079936958892]
  PHRIT is a novel approach for parametric hand mesh modeling with an implicit template. Our method represents deformable hand shapes using signed distance fields (SDFs) with part-based shape priors. We evaluate PHRIT on multiple downstream tasks, including skeleton-driven hand reconstruction, shapes from point clouds, and single-view 3D reconstruction.
  arXiv  Detail & Related papers  (2023-09-26T13:22:33Z)
• MeMaHand: Exploiting Mesh-Mano Interaction for Single Image Two-Hand Reconstruction [19.82874341207336]
  We propose to reconstruct meshes and estimate MANO parameters of two hands from a single RGB image simultaneously. MMIB consists of one graph residual block to aggregate local information and two transformer encoders to model long-range dependencies. Experiments on the InterHand2.6M benchmark demonstrate promising results over the state-of-the-art hand reconstruction methods.
  arXiv  Detail & Related papers  (2023-03-28T04:06:02Z)
• AlignSDF: Pose-Aligned Signed Distance Fields for Hand-Object Reconstruction [76.12874759788298]
  We propose a joint learning framework that disentangles the pose and the shape. We show that such aligned SDFs better focus on reconstructing shape details and improve reconstruction accuracy both for hands and objects.
  arXiv  Detail & Related papers  (2022-07-26T13:58:59Z)
• Towards Accurate Alignment in Real-time 3D Hand-Mesh Reconstruction [57.3636347704271]
  3D hand-mesh reconstruction from RGB images facilitates many applications, including augmented reality (AR) This paper presents a novel pipeline by decoupling the hand-mesh reconstruction task into three stages. We can promote high-quality finger-level mesh-image alignment and drive the models together to deliver real-time predictions.
  arXiv  Detail & Related papers  (2021-09-03T20:42:01Z)
• PaMIR: Parametric Model-Conditioned Implicit Representation for Image-based Human Reconstruction [67.08350202974434]
  We propose Parametric Model-Conditioned Implicit Representation (PaMIR), which combines the parametric body model with the free-form deep implicit function. We show that our method achieves state-of-the-art performance for image-based 3D human reconstruction in the cases of challenging poses and clothing types.
  arXiv  Detail & Related papers  (2020-07-08T02:26:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.