WiLoR: End-to-end 3D Hand Localization and Reconstruction in-the-wild

• URL: http://arxiv.org/abs/2409.12259v1
• Date: Wed, 18 Sep 2024 18:46:51 GMT
• Title: WiLoR: End-to-end 3D Hand Localization and Reconstruction in-the-wild
• Authors: Rolandos Alexandros Potamias, Jinglei Zhang, Jiankang Deng, Stefanos Zafeiriou,
• Abstract summary: We present a data-driven pipeline for efficient multi-hand reconstruction in the wild. The proposed pipeline is composed of two components: a real-time fully convolutional hand localization and a high-fidelity transformer-based 3D hand reconstruction model. Our approach outperforms previous methods in both efficiency and accuracy on popular 2D and 3D benchmarks.
• Score: 53.288327629960364
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: In recent years, 3D hand pose estimation methods have garnered significant attention due to their extensive applications in human-computer interaction, virtual reality, and robotics. In contrast, there has been a notable gap in hand detection pipelines, posing significant challenges in constructing effective real-world multi-hand reconstruction systems. In this work, we present a data-driven pipeline for efficient multi-hand reconstruction in the wild. The proposed pipeline is composed of two components: a real-time fully convolutional hand localization and a high-fidelity transformer-based 3D hand reconstruction model. To tackle the limitations of previous methods and build a robust and stable detection network, we introduce a large-scale dataset with over than 2M in-the-wild hand images with diverse lighting, illumination, and occlusion conditions. Our approach outperforms previous methods in both efficiency and accuracy on popular 2D and 3D benchmarks. Finally, we showcase the effectiveness of our pipeline to achieve smooth 3D hand tracking from monocular videos, without utilizing any temporal components. Code, models, and dataset are available https://rolpotamias.github.io/WiLoR.

Related papers

• Reconstructing Hands in 3D with Transformers [64.15390309553892]
  We present an approach that can reconstruct hands in 3D from monocular input. Our approach for Hand Mesh Recovery, HaMeR, follows a fully transformer-based architecture and can analyze hands with significantly increased accuracy and robustness compared to previous work.
  arXiv  Detail & Related papers  (2023-12-08T18:59:07Z)
• DiffuPose: Monocular 3D Human Pose Estimation via Denoising Diffusion Probabilistic Model [25.223801390996435]
  This paper focuses on reconstructing a 3D pose from a single 2D keypoint detection. We build a novel diffusion-based framework to effectively sample diverse 3D poses from an off-the-shelf 2D detector. We evaluate our method on the widely adopted Human3.6M and HumanEva-I datasets.
  arXiv  Detail & Related papers  (2022-12-06T07:22:20Z)
• End-to-end Weakly-supervised Single-stage Multiple 3D Hand Mesh Reconstruction from a Single RGB Image [9.238322841389994]
  We propose a single-stage pipeline for multi-hand reconstruction. Specifically, we design a multi-head auto-encoder structure, where each head network shares the same feature map and outputs the hand center, pose and texture. Our method outperforms the state-of-the-art model-based methods in both weakly-supervised and fully-supervised manners.
  arXiv  Detail & Related papers  (2022-04-18T03:57:14Z)
• RGB2Hands: Real-Time Tracking of 3D Hand Interactions from Monocular RGB Video [76.86512780916827]
  We present the first real-time method for motion capture of skeletal pose and 3D surface geometry of hands from a single RGB camera. In order to address the inherent depth ambiguities in RGB data, we propose a novel multi-task CNN. We experimentally verify the individual components of our RGB two-hand tracking and 3D reconstruction pipeline.
  arXiv  Detail & Related papers  (2021-06-22T12:53:56Z)
• Synthetic Training for Monocular Human Mesh Recovery [100.38109761268639]
  This paper aims to estimate 3D mesh of multiple body parts with large-scale differences from a single RGB image. The main challenge is lacking training data that have complete 3D annotations of all body parts in 2D images. We propose a depth-to-scale (D2S) projection to incorporate the depth difference into the projection function to derive per-joint scale variants.
  arXiv  Detail & Related papers  (2020-10-27T03:31:35Z)
• BiHand: Recovering Hand Mesh with Multi-stage Bisected Hourglass Networks [37.65510556305611]
  We introduce an end-to-end learnable model, BiHand, which consists of three cascaded stages, namely 2D seeding stage, 3D lifting stage, and mesh generation stage. At the output of BiHand, the full hand mesh will be recovered using the joint rotations and shape parameters predicted from the network. Our model can achieve superior accuracy in comparison with state-of-the-art methods, and can produce appealing 3D hand meshes in several severe conditions.
  arXiv  Detail & Related papers  (2020-08-12T03:13:17Z)
• Two-hand Global 3D Pose Estimation Using Monocular RGB [0.0]
  We tackle the challenging task of estimating global 3D joint locations for both hands via only monocular RGB input images. We propose a novel multi-stage convolutional neural network based pipeline that accurately segments and locates the hands. We present the first work that achieves accurate global 3D hand tracking on both hands using RGB-only inputs.
  arXiv  Detail & Related papers  (2020-06-01T23:53:52Z)
• Exemplar Fine-Tuning for 3D Human Model Fitting Towards In-the-Wild 3D Human Pose Estimation [107.07047303858664]
  Large-scale human datasets with 3D ground-truth annotations are difficult to obtain in the wild. We address this problem by augmenting existing 2D datasets with high-quality 3D pose fits. The resulting annotations are sufficient to train from scratch 3D pose regressor networks that outperform the current state-of-the-art on in-the-wild benchmarks.
  arXiv  Detail & Related papers  (2020-04-07T20:21:18Z)
• Lightweight Multi-View 3D Pose Estimation through Camera-Disentangled Representation [57.11299763566534]
  We present a solution to recover 3D pose from multi-view images captured with spatially calibrated cameras. We exploit 3D geometry to fuse input images into a unified latent representation of pose, which is disentangled from camera view-points. Our architecture then conditions the learned representation on camera projection operators to produce accurate per-view 2d detections.
  arXiv  Detail & Related papers  (2020-04-05T12:52:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.