HandOccNet: Occlusion-Robust 3D Hand Mesh Estimation Network

• URL: http://arxiv.org/abs/2203.14564v1
• Date: Mon, 28 Mar 2022 08:12:16 GMT
• Title: HandOccNet: Occlusion-Robust 3D Hand Mesh Estimation Network
• Authors: JoonKyu Park, Yeonguk Oh, Gyeongsik Moon, Hongsuk Choi, Kyoung Mu Lee
• Abstract summary: We propose a novel 3D hand mesh estimation network HandOccNet. By injecting the hand information to the occluded region, our HandOccNet reaches the state-of-the-art performance on 3D hand mesh benchmarks.
• Score: 57.206129938611454
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Hands are often severely occluded by objects, which makes 3D hand mesh estimation challenging. Previous works often have disregarded information at occluded regions. However, we argue that occluded regions have strong correlations with hands so that they can provide highly beneficial information for complete 3D hand mesh estimation. Thus, in this work, we propose a novel 3D hand mesh estimation network HandOccNet, that can fully exploits the information at occluded regions as a secondary means to enhance image features and make it much richer. To this end, we design two successive Transformer-based modules, called feature injecting transformer (FIT) and self- enhancing transformer (SET). FIT injects hand information into occluded region by considering their correlation. SET refines the output of FIT by using a self-attention mechanism. By injecting the hand information to the occluded region, our HandOccNet reaches the state-of-the-art performance on 3D hand mesh benchmarks that contain challenging hand-object occlusions. The codes are available in: https://github.com/namepllet/HandOccNet.

Related papers

• Two Hands Are Better Than One: Resolving Hand to Hand Intersections via Occupancy Networks [33.9893684177763]
  Self-occlusions and finger articulation pose a significant problem to estimation. We exploit an occupancy network that represents the hand's volume as a continuous manifold. We design an intersection loss function to minimize the likelihood of hand-to-point intersections.
  arXiv  Detail & Related papers  (2024-04-08T11:32:26Z)
• HandBooster: Boosting 3D Hand-Mesh Reconstruction by Conditional Synthesis and Sampling of Hand-Object Interactions [68.28684509445529]
  We present HandBooster, a new approach to uplift the data diversity and boost the 3D hand-mesh reconstruction performance. First, we construct versatile content-aware conditions to guide a diffusion model to produce realistic images with diverse hand appearances, poses, views, and backgrounds. Then, we design a novel condition creator based on our similarity-aware distribution sampling strategies to deliberately find novel and realistic interaction poses that are distinctive from the training set.
  arXiv  Detail & Related papers  (2024-03-27T13:56:08Z)
• HandGCAT: Occlusion-Robust 3D Hand Mesh Reconstruction from Monocular Images [9.554136347258057]
  We propose a robust and accurate method for reconstructing 3D hand mesh from monocular images. HandGCAT can fully exploit hand prior as compensation information to enhance occluded region features.
  arXiv  Detail & Related papers  (2024-02-27T03:40:43Z)
• HOISDF: Constraining 3D Hand-Object Pose Estimation with Global Signed Distance Fields [96.04424738803667]
  HOISDF is a guided hand-object pose estimation network. It exploits hand and object SDFs to provide a global, implicit representation over the complete reconstruction volume. We show that HOISDF achieves state-of-the-art results on hand-object pose estimation benchmarks.
  arXiv  Detail & Related papers  (2024-02-26T22:48:37Z)
• Towards Accurate Alignment in Real-time 3D Hand-Mesh Reconstruction [57.3636347704271]
  3D hand-mesh reconstruction from RGB images facilitates many applications, including augmented reality (AR) This paper presents a novel pipeline by decoupling the hand-mesh reconstruction task into three stages. We can promote high-quality finger-level mesh-image alignment and drive the models together to deliver real-time predictions.
  arXiv  Detail & Related papers  (2021-09-03T20:42:01Z)
• DeepHandMesh: A Weakly-supervised Deep Encoder-Decoder Framework for High-fidelity Hand Mesh Modeling [75.69585456580505]
  DeepHandMesh is a weakly-supervised deep encoder-decoder framework for high-fidelity hand mesh modeling. We show that our system can also be applied successfully to the 3D hand mesh estimation from general images.
  arXiv  Detail & Related papers  (2020-08-19T00:59:51Z)
• BiHand: Recovering Hand Mesh with Multi-stage Bisected Hourglass Networks [37.65510556305611]
  We introduce an end-to-end learnable model, BiHand, which consists of three cascaded stages, namely 2D seeding stage, 3D lifting stage, and mesh generation stage. At the output of BiHand, the full hand mesh will be recovered using the joint rotations and shape parameters predicted from the network. Our model can achieve superior accuracy in comparison with state-of-the-art methods, and can produce appealing 3D hand meshes in several severe conditions.
  arXiv  Detail & Related papers  (2020-08-12T03:13:17Z)
• Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wild [59.158592526006814]
  We train our network by gathering a large-scale dataset of hand action in YouTube videos. Our weakly-supervised mesh convolutions-based system largely outperforms state-of-the-art methods, even halving the errors on the in the wild benchmark.
  arXiv  Detail & Related papers  (2020-04-04T14:35:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.