PlenOctrees for Real-time Rendering of Neural Radiance Fields

• URL: http://arxiv.org/abs/2103.14024v1
• Date: Thu, 25 Mar 2021 17:59:06 GMT
• Title: PlenOctrees for Real-time Rendering of Neural Radiance Fields
• Authors: Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, Angjoo Kanazawa
• Abstract summary: We introduce a method to render Neural Radiance Fields (NeRFs) in real time using PlenOctrees, an octree-based 3D representation. Our method can render 800x800 images at more than 150 FPS, which is over 3000 times faster than conventional NeRFs.
• Score: 35.58442869498845
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce a method to render Neural Radiance Fields (NeRFs) in real time using PlenOctrees, an octree-based 3D representation which supports view-dependent effects. Our method can render 800x800 images at more than 150 FPS, which is over 3000 times faster than conventional NeRFs. We do so without sacrificing quality while preserving the ability of NeRFs to perform free-viewpoint rendering of scenes with arbitrary geometry and view-dependent effects. Real-time performance is achieved by pre-tabulating the NeRF into a PlenOctree. In order to preserve view-dependent effects such as specularities, we factorize the appearance via closed-form spherical basis functions. Specifically, we show that it is possible to train NeRFs to predict a spherical harmonic representation of radiance, removing the viewing direction as an input to the neural network. Furthermore, we show that PlenOctrees can be directly optimized to further minimize the reconstruction loss, which leads to equal or better quality compared to competing methods. Moreover, this octree optimization step can be used to reduce the training time, as we no longer need to wait for the NeRF training to converge fully. Our real-time neural rendering approach may potentially enable new applications such as 6-DOF industrial and product visualizations, as well as next generation AR/VR systems. PlenOctrees are amenable to in-browser rendering as well; please visit the project page for the interactive online demo, as well as video and code: https://alexyu.net/plenoctrees

Related papers

• NeRF-Casting: Improved View-Dependent Appearance with Consistent Reflections [57.63028964831785]
  Recent works have improved NeRF's ability to render detailed specular appearance of distant environment illumination, but are unable to synthesize consistent reflections of closer content. We address these issues with an approach based on ray tracing. Instead of querying an expensive neural network for the outgoing view-dependent radiance at points along each camera ray, our model casts rays from these points and traces them through the NeRF representation to render feature vectors.
  arXiv  Detail & Related papers  (2024-05-23T17:59:57Z)
• PyNeRF: Pyramidal Neural Radiance Fields [51.25406129834537]
  We propose a simple modification to grid-based models by training model heads at different spatial grid resolutions. At render time, we simply use coarser grids to render samples that cover larger volumes. Compared to Mip-NeRF, we reduce error rates by 20% while training over 60x faster.
  arXiv  Detail & Related papers  (2023-11-30T23:52:46Z)
• Reconstructive Latent-Space Neural Radiance Fields for Efficient 3D Scene Representations [34.836151514152746]
  In this work, we investigate combining an autoencoder with a NeRF, in which latent features are rendered and then convolutionally decoded. The resulting latent-space NeRF can produce novel views with higher quality than standard colour-space NeRFs. We can control the tradeoff between efficiency and image quality by shrinking the AE architecture, achieving over 13 times faster rendering with only a small drop in performance.
  arXiv  Detail & Related papers  (2023-10-27T03:52:08Z)
• SimpleNeRF: Regularizing Sparse Input Neural Radiance Fields with Simpler Solutions [6.9980855647933655]
  supervising the depth estimated by the NeRF helps train it effectively with fewer views. We design augmented models that encourage simpler solutions by exploring the role of positional encoding and view-dependent radiance. We achieve state-of-the-art view-synthesis performance on two popular datasets by employing the above regularizations.
  arXiv  Detail & Related papers  (2023-09-07T18:02:57Z)
• SurfelNeRF: Neural Surfel Radiance Fields for Online Photorealistic Reconstruction of Indoor Scenes [17.711755550841385]
  SLAM-based methods can reconstruct 3D scene geometry progressively in real time but can not render photorealistic results. NeRF-based methods produce promising novel view synthesis results, their long offline optimization time and lack of geometric constraints pose challenges to efficiently handling online input. We introduce SurfelNeRF, a variant of neural radiance field which employs a flexible and scalable neural surfel representation to store geometric attributes and extracted appearance features from input images.
  arXiv  Detail & Related papers  (2023-04-18T13:11:49Z)
• R2L: Distilling Neural Radiance Field to Neural Light Field for Efficient Novel View Synthesis [76.07010495581535]
  Rendering a single pixel requires querying the Neural Radiance Field network hundreds of times. NeLF presents a more straightforward representation over NeRF in novel view. We show the key to successfully learning a deep NeLF network is to have sufficient data.
  arXiv  Detail & Related papers  (2022-03-31T17:57:05Z)
• NeRFusion: Fusing Radiance Fields for Large-Scale Scene Reconstruction [50.54946139497575]
  We propose NeRFusion, a method that combines the advantages of NeRF and TSDF-based fusion techniques to achieve efficient large-scale reconstruction and photo-realistic rendering. We demonstrate that NeRFusion achieves state-of-the-art quality on both large-scale indoor and small-scale object scenes, with substantially faster reconstruction than NeRF and other recent methods.
  arXiv  Detail & Related papers  (2022-03-21T18:56:35Z)
• Mega-NeRF: Scalable Construction of Large-Scale NeRFs for Virtual Fly-Throughs [54.41204057689033]
  We explore how to leverage neural fields (NeRFs) to build interactive 3D environments from large-scale visual captures spanning buildings or even multiple city blocks collected primarily from drone data. In contrast to the single object scenes against which NeRFs have been traditionally evaluated, this setting poses multiple challenges. We introduce a simple clustering algorithm that partitions training images (or rather pixels) into different NeRF submodules that can be trained in parallel.
  arXiv  Detail & Related papers  (2021-12-20T17:40:48Z)
• BARF: Bundle-Adjusting Neural Radiance Fields [104.97810696435766]
  We propose Bundle-Adjusting Neural Radiance Fields (BARF) for training NeRF from imperfect camera poses. BARF can effectively optimize the neural scene representations and resolve large camera pose misalignment at the same time. This enables view synthesis and localization of video sequences from unknown camera poses, opening up new avenues for visual localization systems.
  arXiv  Detail & Related papers  (2021-04-13T17:59:51Z)
• Baking Neural Radiance Fields for Real-Time View Synthesis [41.07052395570522]
  We present a method to train a NeRF, then precompute and store (i.e. "bake") it as a novel representation called a Sparse Neural Radiance Grid (SNeRG) The resulting scene representation retains NeRF's ability to render fine geometric details and view-dependent appearance, is compact, and can be rendered in real-time.
  arXiv  Detail & Related papers  (2021-03-26T17:59:52Z)
• FastNeRF: High-Fidelity Neural Rendering at 200FPS [17.722927021159393]
  We propose FastNeRF, a system capable of rendering high fidelity images at 200Hz on a high-end consumer GPU. The proposed method is 3000 times faster than the original NeRF algorithm and at least an order of magnitude faster than existing work on accelerating NeRF.
  arXiv  Detail & Related papers  (2021-03-18T17:09:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.