HandOcc: NeRF-based Hand Rendering with Occupancy Networks

• URL: http://arxiv.org/abs/2505.02079v1
• Date: Sun, 04 May 2025 12:06:54 GMT
• Title: HandOcc: NeRF-based Hand Rendering with Occupancy Networks
• Authors: Maksym Ivashechkin, Oscar Mendez, Richard Bowden,
• Abstract summary: We propose HandOcc, a novel framework for hand rendering based upon occupancy.<n>This paper presents a pipeline for meshless 3D rendering, which we apply to the hands.<n>We achieve state-of-the-art results on the benchmark InterHand2.6M dataset.
• Score: 33.9893684177763
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: We propose HandOcc, a novel framework for hand rendering based upon occupancy. Popular rendering methods such as NeRF are often combined with parametric meshes to provide deformable hand models. However, in doing so, such approaches present a trade-off between the fidelity of the mesh and the complexity and dimensionality of the parametric model. The simplicity of parametric mesh structures is appealing, but the underlying issue is that it binds methods to mesh initialization, making it unable to generalize to objects where a parametric model does not exist. It also means that estimation is tied to mesh resolution and the accuracy of mesh fitting. This paper presents a pipeline for meshless 3D rendering, which we apply to the hands. By providing only a 3D skeleton, the desired appearance is extracted via a convolutional model. We do this by exploiting a NeRF renderer conditioned upon an occupancy-based representation. The approach uses the hand occupancy to resolve hand-to-hand interactions further improving results, allowing fast rendering, and excellent hand appearance transfer. On the benchmark InterHand2.6M dataset, we achieved state-of-the-art results.

Related papers

• Two Hands Are Better Than One: Resolving Hand to Hand Intersections via Occupancy Networks [33.9893684177763]
  Self-occlusions and finger articulation pose a significant problem to estimation. We exploit an occupancy network that represents the hand's volume as a continuous manifold. We design an intersection loss function to minimize the likelihood of hand-to-point intersections.
  arXiv  Detail & Related papers  (2024-04-08T11:32:26Z)
• Monocular 3D Hand Mesh Recovery via Dual Noise Estimation [47.82179706128616]
  We introduce a dual noise estimation method to generate meshes that are aligned with the image well. Our method achieves state-of-the-art performance on a large-scale Interhand2.6M dataset.
  arXiv  Detail & Related papers  (2023-12-26T07:21:01Z)
• Overcoming the Trade-off Between Accuracy and Plausibility in 3D Hand Shape Reconstruction [62.96478903239799]
  Direct mesh fitting for 3D hand shape reconstruction is highly accurate. However, the reconstructed meshes are prone to artifacts and do not appear as plausible hand shapes. We introduce a novel weakly-supervised hand shape estimation framework that integrates non-parametric mesh fitting with MANO model in an end-to-end fashion.
  arXiv  Detail & Related papers  (2023-05-01T03:38:01Z)
• HandOccNet: Occlusion-Robust 3D Hand Mesh Estimation Network [57.206129938611454]
  We propose a novel 3D hand mesh estimation network HandOccNet. By injecting the hand information to the occluded region, our HandOccNet reaches the state-of-the-art performance on 3D hand mesh benchmarks.
  arXiv  Detail & Related papers  (2022-03-28T08:12:16Z)
• Mesh Draping: Parametrization-Free Neural Mesh Transfer [92.55503085245304]
  Mesh Draping is a neural method for transferring existing mesh structure from one shape to another. We show that by leveraging gradually increasing frequencies to guide the neural optimization, we are able to achieve stable and high quality mesh transfer.
  arXiv  Detail & Related papers  (2021-10-11T17:24:52Z)
• Towards Accurate Alignment in Real-time 3D Hand-Mesh Reconstruction [57.3636347704271]
  3D hand-mesh reconstruction from RGB images facilitates many applications, including augmented reality (AR) This paper presents a novel pipeline by decoupling the hand-mesh reconstruction task into three stages. We can promote high-quality finger-level mesh-image alignment and drive the models together to deliver real-time predictions.
  arXiv  Detail & Related papers  (2021-09-03T20:42:01Z)
• HandFoldingNet: A 3D Hand Pose Estimation Network Using Multiscale-Feature Guided Folding of a 2D Hand Skeleton [4.1954750695245835]
  This paper proposes HandFoldingNet, an accurate and efficient hand pose estimator. The proposed model utilizes a folding-based decoder that folds a given 2D hand skeleton into the corresponding joint coordinates. Experimental results show that the proposed model outperforms the existing methods on three hand pose benchmark datasets.
  arXiv  Detail & Related papers  (2021-08-12T05:52:44Z)
• HandVoxNet++: 3D Hand Shape and Pose Estimation using Voxel-Based Neural Networks [71.09275975580009]
  HandVoxNet++ is a voxel-based deep network with 3D and graph convolutions trained in a fully supervised manner. HandVoxNet++ relies on two hand shape representations. The first one is the 3D voxelized grid of hand shape, which does not preserve the mesh topology. We combine the advantages of both representations by aligning the hand surface to the voxelized hand shape either with a new neural Graph-Convolutions-based Mesh Registration (GCN-MeshReg) or classical segment-wise Non-Rigid Gravitational Approach (NRGA++) which
  arXiv  Detail & Related papers  (2021-07-02T17:59:54Z)
• Im2Mesh GAN: Accurate 3D Hand Mesh Recovery from a Single RGB Image [31.371190180801452]
  We show that the hand mesh can be learned directly from the input image. We propose a new type of GAN called Im2Mesh GAN to learn the mesh through end-to-end adversarial training.
  arXiv  Detail & Related papers  (2021-01-27T07:38:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.