Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wild

• URL: http://arxiv.org/abs/2004.01946v1
• Date: Sat, 4 Apr 2020 14:35:37 GMT
• Title: Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wild
• Authors: Dominik Kulon, Riza Alp G\"uler, Iasonas Kokkinos, Michael Bronstein, Stefanos Zafeiriou
• Abstract summary: We train our network by gathering a large-scale dataset of hand action in YouTube videos. Our weakly-supervised mesh convolutions-based system largely outperforms state-of-the-art methods, even halving the errors on the in the wild benchmark.
• Score: 59.158592526006814
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce a simple and effective network architecture for monocular 3D hand pose estimation consisting of an image encoder followed by a mesh convolutional decoder that is trained through a direct 3D hand mesh reconstruction loss. We train our network by gathering a large-scale dataset of hand action in YouTube videos and use it as a source of weak supervision. Our weakly-supervised mesh convolutions-based system largely outperforms state-of-the-art methods, even halving the errors on the in the wild benchmark. The dataset and additional resources are available at https://arielai.com/mesh_hands.

Related papers

• WiLoR: End-to-end 3D Hand Localization and Reconstruction in-the-wild [53.288327629960364]
  We present a data-driven pipeline for efficient multi-hand reconstruction in the wild. The proposed pipeline is composed of two components: a real-time fully convolutional hand localization and a high-fidelity transformer-based 3D hand reconstruction model. Our approach outperforms previous methods in both efficiency and accuracy on popular 2D and 3D benchmarks.
  arXiv  Detail & Related papers  (2024-09-18T18:46:51Z)
• Fine-Grained Multi-View Hand Reconstruction Using Inverse Rendering [11.228453237603834]
  We present a novel fine-grained multi-view hand mesh reconstruction method that leverages inverse rendering to restore hand poses and intricate details. We also introduce a novel Hand Albedo and Mesh (HAM) optimization module to refine both the hand mesh and textures. Our proposed approach outperforms the state-of-the-art methods on both reconstruction accuracy and rendering quality.
  arXiv  Detail & Related papers  (2024-07-08T07:28:24Z)
• HandOccNet: Occlusion-Robust 3D Hand Mesh Estimation Network [57.206129938611454]
  We propose a novel 3D hand mesh estimation network HandOccNet. By injecting the hand information to the occluded region, our HandOccNet reaches the state-of-the-art performance on 3D hand mesh benchmarks.
  arXiv  Detail & Related papers  (2022-03-28T08:12:16Z)
• Towards Accurate Alignment in Real-time 3D Hand-Mesh Reconstruction [57.3636347704271]
  3D hand-mesh reconstruction from RGB images facilitates many applications, including augmented reality (AR) This paper presents a novel pipeline by decoupling the hand-mesh reconstruction task into three stages. We can promote high-quality finger-level mesh-image alignment and drive the models together to deliver real-time predictions.
  arXiv  Detail & Related papers  (2021-09-03T20:42:01Z)
• Parallel mesh reconstruction streams for pose estimation of interacting hands [2.0305676256390934]
  We present a new multi-stream 3D mesh reconstruction network (MSMR-Net) for hand pose estimation from a single RGB image. Our model consists of an image encoder followed by a mesh-convolution decoder composed of connected graph convolution layers.
  arXiv  Detail & Related papers  (2021-04-25T10:14:15Z)
• Model-based 3D Hand Reconstruction via Self-Supervised Learning [72.0817813032385]
  Reconstructing a 3D hand from a single-view RGB image is challenging due to various hand configurations and depth ambiguity. We propose S2HAND, a self-supervised 3D hand reconstruction network that can jointly estimate pose, shape, texture, and the camera viewpoint. For the first time, we demonstrate the feasibility of training an accurate 3D hand reconstruction network without relying on manual annotations.
  arXiv  Detail & Related papers  (2021-03-22T10:12:43Z)
• Im2Mesh GAN: Accurate 3D Hand Mesh Recovery from a Single RGB Image [31.371190180801452]
  We show that the hand mesh can be learned directly from the input image. We propose a new type of GAN called Im2Mesh GAN to learn the mesh through end-to-end adversarial training.
  arXiv  Detail & Related papers  (2021-01-27T07:38:01Z)
• DeepHandMesh: A Weakly-supervised Deep Encoder-Decoder Framework for High-fidelity Hand Mesh Modeling [75.69585456580505]
  DeepHandMesh is a weakly-supervised deep encoder-decoder framework for high-fidelity hand mesh modeling. We show that our system can also be applied successfully to the 3D hand mesh estimation from general images.
  arXiv  Detail & Related papers  (2020-08-19T00:59:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.