EG-HumanNeRF: Efficient Generalizable Human NeRF Utilizing Human Prior for Sparse View

• URL: http://arxiv.org/abs/2410.12242v1
• Date: Wed, 16 Oct 2024 05:08:00 GMT
• Title: EG-HumanNeRF: Efficient Generalizable Human NeRF Utilizing Human Prior for Sparse View
• Authors: Zhaorong Wang, Yoshihiro Kanamori, Yuki Endo,
• Abstract summary: Generalizable neural radiance field (NeRF) enables neural-based digital human rendering without per-scene retraining. We propose a generalizable human NeRF framework that achieves high-quality and real-time rendering with sparse input views.
• Score: 2.11923215233494
• License:
• Abstract: Generalizable neural radiance field (NeRF) enables neural-based digital human rendering without per-scene retraining. When combined with human prior knowledge, high-quality human rendering can be achieved even with sparse input views. However, the inference of these methods is still slow, as a large number of neural network queries on each ray are required to ensure the rendering quality. Moreover, occluded regions often suffer from artifacts, especially when the input views are sparse. To address these issues, we propose a generalizable human NeRF framework that achieves high-quality and real-time rendering with sparse input views by extensively leveraging human prior knowledge. We accelerate the rendering with a two-stage sampling reduction strategy: first constructing boundary meshes around the human geometry to reduce the number of ray samples for sampling guidance regression, and then volume rendering using fewer guided samples. To improve rendering quality, especially in occluded regions, we propose an occlusion-aware attention mechanism to extract occlusion information from the human priors, followed by an image space refinement network to improve rendering quality. Furthermore, for volume rendering, we adopt a signed ray distance function (SRDF) formulation, which allows us to propose an SRDF loss at every sample position to improve the rendering quality further. Our experiments demonstrate that our method outperforms the state-of-the-art methods in rendering quality and has a competitive rendering speed compared with speed-prioritized novel view synthesis methods.

Related papers

• LIFe-GoM: Generalizable Human Rendering with Learned Iterative Feedback Over Multi-Resolution Gaussians-on-Mesh [102.24454703207194]
  Generalizable rendering of an animatable human avatar from sparse inputs relies on data priors and inductive biases extracted from training on large data. We propose an iterative feedback update framework, which successively improves the canonical human shape representation during reconstruction. Our approach reconstructs an animatable representation from sparse inputs in less than 1s, renders views with 95.1FPS at $1024 times 1024$, and achieves PSNR/LPIPS*/FID of 24.65/110.82/51.27 on THuman2.0.
  arXiv  Detail & Related papers  (2025-02-13T18:59:19Z)
• Few-Shot Multi-Human Neural Rendering Using Geometry Constraints [8.819403814092865]
  We present a method for recovering the shape and radiance of a scene consisting of multiple people given solely a few images. Existing approaches using implicit neural representations have achieved impressive results that deliver accurate geometry and appearance. We propose a neural implicit reconstruction method that addresses the inherent challenges of this task through the following contributions.
  arXiv  Detail & Related papers  (2025-02-11T00:10:58Z)
• VoxNeRF: Bridging Voxel Representation and Neural Radiance Fields for Enhanced Indoor View Synthesis [73.50359502037232]
  VoxNeRF is a novel approach to enhance the quality and efficiency of neural indoor reconstruction and novel view synthesis. We propose an efficient voxel-guided sampling technique that allocates computational resources to selectively the most relevant segments of rays. Our approach is validated with extensive experiments on ScanNet and ScanNet++.
  arXiv  Detail & Related papers  (2023-11-09T11:32:49Z)
• DARF: Depth-Aware Generalizable Neural Radiance Field [51.29437249009986]
  We propose the Depth-Aware Generalizable Neural Radiance Field (DARF) with a Depth-Aware Dynamic Sampling (DADS) strategy. Our framework infers the unseen scenes on both pixel level and geometry level with only a few input images. Compared with state-of-the-art generalizable NeRF methods, DARF reduces samples by 50%, while improving rendering quality and depth estimation.
  arXiv  Detail & Related papers  (2022-12-05T14:00:59Z)
• InfoNeRF: Ray Entropy Minimization for Few-Shot Neural Volume Rendering [55.70938412352287]
  We present an information-theoretic regularization technique for few-shot novel view synthesis based on neural implicit representation. The proposed approach minimizes potential reconstruction inconsistency that happens due to insufficient viewpoints. We achieve consistently improved performance compared to existing neural view synthesis methods by large margins on multiple standard benchmarks.
  arXiv  Detail & Related papers  (2021-12-31T11:56:01Z)
• Geometry-Guided Progressive NeRF for Generalizable and Efficient Neural Human Rendering [139.159534903657]
  We develop a generalizable and efficient Neural Radiance Field (NeRF) pipeline for high-fidelity free-viewpoint human body details. To better tackle self-occlusion, we devise a geometry-guided multi-view feature integration approach. For achieving higher rendering efficiency, we introduce a geometry-guided progressive rendering pipeline.
  arXiv  Detail & Related papers  (2021-12-08T14:42:10Z)
• Few-shot Neural Human Performance Rendering from Sparse RGBD Videos [40.20382131461408]
  Recent neural rendering approaches for human activities achieve remarkable view rendering results, but still rely on input views dense training. We propose a fewshot neural rendering approach (FNHR) from only RGBD inputs to generate photoview free free viewpoint results.
  arXiv  Detail & Related papers  (2021-07-14T06:28:16Z)
• NeRF in detail: Learning to sample for view synthesis [104.75126790300735]
  Neural radiance fields (NeRF) methods have demonstrated impressive novel view synthesis. In this work we address a clear limitation of the vanilla coarse-to-fine approach -- that it is based on a performance and not trained end-to-end for the task at hand. We introduce a differentiable module that learns to propose samples and their importance for the fine network, and consider and compare multiple alternatives for its neural architecture.
  arXiv  Detail & Related papers  (2021-06-09T17:59:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.