HandNeRF: Neural Radiance Fields for Animatable Interacting Hands
- URL: http://arxiv.org/abs/2303.13825v1
- Date: Fri, 24 Mar 2023 06:19:19 GMT
- Title: HandNeRF: Neural Radiance Fields for Animatable Interacting Hands
- Authors: Zhiyang Guo, Wengang Zhou, Min Wang, Li Li, Houqiang Li
- Abstract summary: We propose a novel framework to reconstruct accurate appearance and geometry with neural radiance fields (NeRF) for interacting hands.
We conduct extensive experiments to verify the merits of our proposed HandNeRF and report a series of state-of-the-art results.
- Score: 122.32855646927013
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We propose a novel framework to reconstruct accurate appearance and geometry
with neural radiance fields (NeRF) for interacting hands, enabling the
rendering of photo-realistic images and videos for gesture animation from
arbitrary views. Given multi-view images of a single hand or interacting hands,
an off-the-shelf skeleton estimator is first employed to parameterize the hand
poses. Then we design a pose-driven deformation field to establish
correspondence from those different poses to a shared canonical space, where a
pose-disentangled NeRF for one hand is optimized. Such unified modeling
efficiently complements the geometry and texture cues in rarely-observed areas
for both hands. Meanwhile, we further leverage the pose priors to generate
pseudo depth maps as guidance for occlusion-aware density learning. Moreover, a
neural feature distillation method is proposed to achieve cross-domain
alignment for color optimization. We conduct extensive experiments to verify
the merits of our proposed HandNeRF and report a series of state-of-the-art
results both qualitatively and quantitatively on the large-scale InterHand2.6M
dataset.
Related papers
- Fine-Grained Multi-View Hand Reconstruction Using Inverse Rendering [11.228453237603834]
We present a novel fine-grained multi-view hand mesh reconstruction method that leverages inverse rendering to restore hand poses and intricate details.
We also introduce a novel Hand Albedo and Mesh (HAM) optimization module to refine both the hand mesh and textures.
Our proposed approach outperforms the state-of-the-art methods on both reconstruction accuracy and rendering quality.
arXiv Detail & Related papers (2024-07-08T07:28:24Z) - NCRF: Neural Contact Radiance Fields for Free-Viewpoint Rendering of
Hand-Object Interaction [19.957593804898064]
We present a novel free-point rendering framework, Neural Contact Radiance Field ( NCRF), to reconstruct hand-object interactions from a sparse set of videos.
We jointly learn these key components where they mutually help and regularize each other with visual and geometric constraints.
Our approach outperforms the current state-of-the-art in terms of both rendering quality and pose estimation accuracy.
arXiv Detail & Related papers (2024-02-08T10:09:12Z) - 3D Visibility-aware Generalizable Neural Radiance Fields for Interacting
Hands [51.305421495638434]
Neural radiance fields (NeRFs) are promising 3D representations for scenes, objects, and humans.
This paper proposes a generalizable visibility-aware NeRF framework for interacting hands.
Experiments on the Interhand2.6M dataset demonstrate that our proposed VA-NeRF outperforms conventional NeRFs significantly.
arXiv Detail & Related papers (2024-01-02T00:42:06Z) - Pose-Free Neural Radiance Fields via Implicit Pose Regularization [117.648238941948]
IR-NeRF is an innovative pose-free neural radiance field (NeRF) that introduces implicit pose regularization to refine pose estimator with unposed real images.
With a collection of 2D images of a specific scene, IR-NeRF constructs a scene codebook that stores scene features and captures the scene-specific pose distribution implicitly as priors.
arXiv Detail & Related papers (2023-08-29T06:14:06Z) - GM-NeRF: Learning Generalizable Model-based Neural Radiance Fields from
Multi-view Images [79.39247661907397]
We introduce an effective framework Generalizable Model-based Neural Radiance Fields to synthesize free-viewpoint images.
Specifically, we propose a geometry-guided attention mechanism to register the appearance code from multi-view 2D images to a geometry proxy.
arXiv Detail & Related papers (2023-03-24T03:32:02Z) - Im2Hands: Learning Attentive Implicit Representation of Interacting
Two-Hand Shapes [58.551154822792284]
Implicit Two Hands (Im2Hands) is the first neural implicit representation of two interacting hands.
Im2Hands can produce fine-grained geometry of two hands with high hand-to-hand and hand-to-image coherency.
We experimentally demonstrate the effectiveness of Im2Hands on two-hand reconstruction in comparison to related methods.
arXiv Detail & Related papers (2023-02-28T06:38:25Z) - Real-time Pose and Shape Reconstruction of Two Interacting Hands With a
Single Depth Camera [79.41374930171469]
We present a novel method for real-time pose and shape reconstruction of two strongly interacting hands.
Our approach combines an extensive list of favorable properties, namely it is marker-less.
We show state-of-the-art results in scenes that exceed the complexity level demonstrated by previous work.
arXiv Detail & Related papers (2021-06-15T11:39:49Z) - Skeleton-aware multi-scale heatmap regression for 2D hand pose
estimation [1.0152838128195467]
We propose a new deep learning-based framework that consists of two main modules.
The former presents a segmentation-based approach to detect the hand skeleton and localize the hand bounding box.
The second module regresses the 2D joint locations through a multi-scale heatmap regression approach.
arXiv Detail & Related papers (2021-05-23T10:23:51Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.