HARP: Personalized Hand Reconstruction from a Monocular RGB Video

• URL: http://arxiv.org/abs/2212.09530v3
• Date: Mon, 3 Jul 2023 21:16:17 GMT
• Title: HARP: Personalized Hand Reconstruction from a Monocular RGB Video
• Authors: Korrawe Karunratanakul, Sergey Prokudin, Otmar Hilliges, Siyu Tang
• Abstract summary: We present HARP, a personalized hand avatar creation approach that takes a short monocular RGB video of a human hand as input. In contrast to the major trend of neural implicit representations, HARP models a hand with a mesh-based parametric hand model. HarP can be directly used in AR/VR applications with real-time rendering capability.
• Score: 37.384221764796095
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present HARP (HAnd Reconstruction and Personalization), a personalized hand avatar creation approach that takes a short monocular RGB video of a human hand as input and reconstructs a faithful hand avatar exhibiting a high-fidelity appearance and geometry. In contrast to the major trend of neural implicit representations, HARP models a hand with a mesh-based parametric hand model, a vertex displacement map, a normal map, and an albedo without any neural components. As validated by our experiments, the explicit nature of our representation enables a truly scalable, robust, and efficient approach to hand avatar creation. HARP is optimized via gradient descent from a short sequence captured by a hand-held mobile phone and can be directly used in AR/VR applications with real-time rendering capability. To enable this, we carefully design and implement a shadow-aware differentiable rendering scheme that is robust to high degree articulations and self-shadowing regularly present in hand motion sequences, as well as challenging lighting conditions. It also generalizes to unseen poses and novel viewpoints, producing photo-realistic renderings of hand animations performing highly-articulated motions. Furthermore, the learned HARP representation can be used for improving 3D hand pose estimation quality in challenging viewpoints. The key advantages of HARP are validated by the in-depth analyses on appearance reconstruction, novel-view and novel pose synthesis, and 3D hand pose refinement. It is an AR/VR-ready personalized hand representation that shows superior fidelity and scalability.

Related papers

• XHand: Real-time Expressive Hand Avatar [9.876680405587745]
  We introduce an expressive hand avatar, named XHand, that is designed to generate hand shape, appearance, and deformations in real-time. XHand is able to recover high-fidelity geometry and texture for hand animations across diverse poses in real-time.
  arXiv  Detail & Related papers  (2024-07-30T17:49:21Z)
• On the Utility of 3D Hand Poses for Action Recognition [36.64538554919222]
  HandFormer is a novel multimodal transformer to efficiently model hand-object interactions. We factorize hand modeling and represent each joint by its short-term trajectories. We achieve new state-of-the-art performance on Assembly101 and H2O with significant improvements in egocentric action recognition.
  arXiv  Detail & Related papers  (2024-03-14T18:52:34Z)
• GaussianAvatar: Towards Realistic Human Avatar Modeling from a Single Video via Animatable 3D Gaussians [51.46168990249278]
  We present an efficient approach to creating realistic human avatars with dynamic 3D appearances from a single video. GustafAvatar is validated on both the public dataset and our collected dataset.
  arXiv  Detail & Related papers  (2023-12-04T18:55:45Z)
• FLARE: Fast Learning of Animatable and Relightable Mesh Avatars [64.48254296523977]
  Our goal is to efficiently learn personalized animatable 3D head avatars from videos that are geometrically accurate, realistic, relightable, and compatible with current rendering systems. We introduce FLARE, a technique that enables the creation of animatable and relightable avatars from a single monocular video.
  arXiv  Detail & Related papers  (2023-10-26T16:13:00Z)
• HandNeRF: Learning to Reconstruct Hand-Object Interaction Scene from a Single RGB Image [41.580285338167315]
  This paper presents a method to learn hand-object interaction prior for reconstructing a 3D hand-object scene from a single RGB image. We use the hand shape to constrain the possible relative configuration of the hand and object geometry. We show that HandNeRF is able to reconstruct hand-object scenes of novel grasp configurations more accurately than comparable methods.
  arXiv  Detail & Related papers  (2023-09-14T17:42:08Z)
• HandNeRF: Neural Radiance Fields for Animatable Interacting Hands [122.32855646927013]
  We propose a novel framework to reconstruct accurate appearance and geometry with neural radiance fields (NeRF) for interacting hands. We conduct extensive experiments to verify the merits of our proposed HandNeRF and report a series of state-of-the-art results.
  arXiv  Detail & Related papers  (2023-03-24T06:19:19Z)
• Hand Avatar: Free-Pose Hand Animation and Rendering from Monocular Video [23.148367696192107]
  We present HandAvatar, a novel representation for hand animation and rendering. HandAvatar can generate smoothly compositional geometry and self-occlusion-aware texture.
  arXiv  Detail & Related papers  (2022-11-23T08:50:03Z)
• HVTR: Hybrid Volumetric-Textural Rendering for Human Avatars [65.82222842213577]
  We propose a novel neural rendering pipeline, which synthesizes virtual human avatars from arbitrary poses efficiently and at high quality. First, we learn to encode articulated human motions on a dense UV manifold of the human body surface. We then leverage the encoded information on the UV manifold to construct a 3D volumetric representation.
  arXiv  Detail & Related papers  (2021-12-19T17:34:15Z)
• Realistic Hands: A Hybrid Model for 3D Hand Reconstruction [14.87768436388877]
  Estimating 3D hand meshes from RGB images robustly is a highly desirable task. Previous methods generally either use parametric 3D hand models or follow a model-free approach. We propose a hybrid approach, utilizing a deep neural network and differential rendering based optimization.
  arXiv  Detail & Related papers  (2021-08-31T17:40:49Z)
• Model-based 3D Hand Reconstruction via Self-Supervised Learning [72.0817813032385]
  Reconstructing a 3D hand from a single-view RGB image is challenging due to various hand configurations and depth ambiguity. We propose S2HAND, a self-supervised 3D hand reconstruction network that can jointly estimate pose, shape, texture, and the camera viewpoint. For the first time, we demonstrate the feasibility of training an accurate 3D hand reconstruction network without relying on manual annotations.
  arXiv  Detail & Related papers  (2021-03-22T10:12:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.