HandNeRF: Neural Radiance Fields for Animatable Interacting Hands

• URL: http://arxiv.org/abs/2303.13825v1
• Date: Fri, 24 Mar 2023 06:19:19 GMT
• Title: HandNeRF: Neural Radiance Fields for Animatable Interacting Hands
• Authors: Zhiyang Guo, Wengang Zhou, Min Wang, Li Li, Houqiang Li
• Abstract summary: We propose a novel framework to reconstruct accurate appearance and geometry with neural radiance fields (NeRF) for interacting hands. We conduct extensive experiments to verify the merits of our proposed HandNeRF and report a series of state-of-the-art results.
• Score: 122.32855646927013
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose a novel framework to reconstruct accurate appearance and geometry with neural radiance fields (NeRF) for interacting hands, enabling the rendering of photo-realistic images and videos for gesture animation from arbitrary views. Given multi-view images of a single hand or interacting hands, an off-the-shelf skeleton estimator is first employed to parameterize the hand poses. Then we design a pose-driven deformation field to establish correspondence from those different poses to a shared canonical space, where a pose-disentangled NeRF for one hand is optimized. Such unified modeling efficiently complements the geometry and texture cues in rarely-observed areas for both hands. Meanwhile, we further leverage the pose priors to generate pseudo depth maps as guidance for occlusion-aware density learning. Moreover, a neural feature distillation method is proposed to achieve cross-domain alignment for color optimization. We conduct extensive experiments to verify the merits of our proposed HandNeRF and report a series of state-of-the-art results both qualitatively and quantitatively on the large-scale InterHand2.6M dataset.

Related papers

• Learning Interaction-aware 3D Gaussian Splatting for One-shot Hand Avatars [47.61442517627826]
  We propose to create animatable avatars for interacting hands with 3D Gaussian Splatting (GS) and single-image inputs. Our proposed method is validated via extensive experiments on the large-scale InterHand2.6M dataset.
  arXiv  Detail & Related papers  (2024-10-11T14:14:51Z)
• Adaptive Multi-Modal Control of Digital Human Hand Synthesis Using a Region-Aware Cycle Loss [12.565642618427844]
  Diffusion models can synthesize images, including the generation of humans in specific poses. Current models face challenges in adequately expressing conditional control for detailed hand pose generation. We propose a novel Region-Aware Cycle Loss (RACL) that enables the diffusion model training to focus on improving the hand region.
  arXiv  Detail & Related papers  (2024-09-13T19:09:19Z)
• NCRF: Neural Contact Radiance Fields for Free-Viewpoint Rendering of Hand-Object Interaction [19.957593804898064]
  We present a novel free-point rendering framework, Neural Contact Radiance Field ( NCRF), to reconstruct hand-object interactions from a sparse set of videos. We jointly learn these key components where they mutually help and regularize each other with visual and geometric constraints. Our approach outperforms the current state-of-the-art in terms of both rendering quality and pose estimation accuracy.
  arXiv  Detail & Related papers  (2024-02-08T10:09:12Z)
• 3D Visibility-aware Generalizable Neural Radiance Fields for Interacting Hands [51.305421495638434]
  Neural radiance fields (NeRFs) are promising 3D representations for scenes, objects, and humans. This paper proposes a generalizable visibility-aware NeRF framework for interacting hands. Experiments on the Interhand2.6M dataset demonstrate that our proposed VA-NeRF outperforms conventional NeRFs significantly.
  arXiv  Detail & Related papers  (2024-01-02T00:42:06Z)
• Pose-Free Neural Radiance Fields via Implicit Pose Regularization [117.648238941948]
  IR-NeRF is an innovative pose-free neural radiance field (NeRF) that introduces implicit pose regularization to refine pose estimator with unposed real images. With a collection of 2D images of a specific scene, IR-NeRF constructs a scene codebook that stores scene features and captures the scene-specific pose distribution implicitly as priors.
  arXiv  Detail & Related papers  (2023-08-29T06:14:06Z)
• GM-NeRF: Learning Generalizable Model-based Neural Radiance Fields from Multi-view Images [79.39247661907397]
  We introduce an effective framework Generalizable Model-based Neural Radiance Fields to synthesize free-viewpoint images. Specifically, we propose a geometry-guided attention mechanism to register the appearance code from multi-view 2D images to a geometry proxy.
  arXiv  Detail & Related papers  (2023-03-24T03:32:02Z)
• Real-time Pose and Shape Reconstruction of Two Interacting Hands With a Single Depth Camera [79.41374930171469]
  We present a novel method for real-time pose and shape reconstruction of two strongly interacting hands. Our approach combines an extensive list of favorable properties, namely it is marker-less. We show state-of-the-art results in scenes that exceed the complexity level demonstrated by previous work.
  arXiv  Detail & Related papers  (2021-06-15T11:39:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.