Mip-NeRF RGB-D: Depth Assisted Fast Neural Radiance Fields

• URL: http://arxiv.org/abs/2205.09351v1
• Date: Thu, 19 May 2022 07:11:42 GMT
• Title: Mip-NeRF RGB-D: Depth Assisted Fast Neural Radiance Fields
• Authors: Arnab Dey, Yassine Ahmine, Andrew I. Comport
• Abstract summary: Neural scene representations, such as neural radiance fields (NeRF), are based on training a multilayer perceptron (MLP) using a set of color images with known poses. An increasing number of devices now produce RGB-D information, which has been shown to be very important for a wide range of tasks. This paper investigates what improvements can be made to these promising implicit representations by incorporating depth information with the color images.
• Score: 0.696125353550498
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Neural scene representations, such as neural radiance fields (NeRF), are based on training a multilayer perceptron (MLP) using a set of color images with known poses. An increasing number of devices now produce RGB-D information, which has been shown to be very important for a wide range of tasks. Therefore, the aim of this paper is to investigate what improvements can be made to these promising implicit representations by incorporating depth information with the color images. In particular, the recently proposed Mip-NeRF approach, which uses conical frustums instead of rays for volume rendering, allows one to account for the varying area of a pixel with distance from the camera center. The proposed method additionally models depth uncertainty. This allows to address major limitations of NeRF-based approaches including improving the accuracy of geometry, reduced artifacts, faster training time, and shortened prediction time. Experiments are performed on well-known benchmark scenes, and comparisons show improved accuracy in scene geometry and photometric reconstruction, while reducing the training time by 3 - 5 times.

Related papers

• Neural Radiance Fields with Torch Units [19.927273454898295]
  Learning-based 3D reconstruction methods are widely used in industrial applications. In this paper, we propose a novel inference pattern that encourages single camera ray possessing more contextual information. To summarize, as a torchlight, a ray in our proposed method rendering a patch of image. Thus, we call the proposed method, Torch-NeRF.
  arXiv  Detail & Related papers  (2024-04-03T10:08:55Z)
• Leveraging Neural Radiance Fields for Uncertainty-Aware Visual Localization [56.95046107046027]
  We propose to leverage Neural Radiance Fields (NeRF) to generate training samples for scene coordinate regression. Despite NeRF's efficiency in rendering, many of the rendered data are polluted by artifacts or only contain minimal information gain.
  arXiv  Detail & Related papers  (2023-10-10T20:11:13Z)
• Fast Monocular Scene Reconstruction with Global-Sparse Local-Dense Grids [84.90863397388776]
  We propose to directly use signed distance function (SDF) in sparse voxel block grids for fast and accurate scene reconstruction without distances. Our globally sparse and locally dense data structure exploits surfaces' spatial sparsity, enables cache-friendly queries, and allows direct extensions to multi-modal data. Experiments show that our approach is 10x faster in training and 100x faster in rendering while achieving comparable accuracy to state-of-the-art neural implicit methods.
  arXiv  Detail & Related papers  (2023-05-22T16:50:19Z)
• DiffusioNeRF: Regularizing Neural Radiance Fields with Denoising Diffusion Models [5.255302402546892]
  We learn a prior over scene geometry and color, using a denoising diffusion model (DDM) We show that, these gradients of logarithms of RGBD patch priors serve to regularize geometry and color of a scene. Evaluations on LLFF, the most relevant dataset, show that our learned prior achieves improved quality in the reconstructed geometry and improved to novel views.
  arXiv  Detail & Related papers  (2023-02-23T18:52:28Z)
• Shakes on a Plane: Unsupervised Depth Estimation from Unstabilized Photography [54.36608424943729]
  We show that in a ''long-burst'', forty-two 12-megapixel RAW frames captured in a two-second sequence, there is enough parallax information from natural hand tremor alone to recover high-quality scene depth. We devise a test-time optimization approach that fits a neural RGB-D representation to long-burst data and simultaneously estimates scene depth and camera motion.
  arXiv  Detail & Related papers  (2022-12-22T18:54:34Z)
• AligNeRF: High-Fidelity Neural Radiance Fields via Alignment-Aware Training [100.33713282611448]
  We conduct the first pilot study on training NeRF with high-resolution data. We propose the corresponding solutions, including marrying the multilayer perceptron with convolutional layers. Our approach is nearly free without introducing obvious training/testing costs.
  arXiv  Detail & Related papers  (2022-11-17T17:22:28Z)
• CLONeR: Camera-Lidar Fusion for Occupancy Grid-aided Neural Representations [77.90883737693325]
  This paper proposes CLONeR, which significantly improves upon NeRF by allowing it to model large outdoor driving scenes observed from sparse input sensor views. This is achieved by decoupling occupancy and color learning within the NeRF framework into separate Multi-Layer Perceptrons (MLPs) trained using LiDAR and camera data, respectively. In addition, this paper proposes a novel method to build differentiable 3D Occupancy Grid Maps (OGM) alongside the NeRF model, and leverage this occupancy grid for improved sampling of points along a ray for rendering in metric space.
  arXiv  Detail & Related papers  (2022-09-02T17:44:50Z)
• PDRF: Progressively Deblurring Radiance Field for Fast and Robust Scene Reconstruction from Blurry Images [75.87721926918874]
  We present Progressively Deblurring Radiance Field (PDRF) PDRF is a novel approach to efficiently reconstruct high quality radiance fields from blurry images. We show that PDRF is 15X faster than previous State-of-The-Art scene reconstruction methods.
  arXiv  Detail & Related papers  (2022-08-17T03:42:29Z)
• RGB-D Neural Radiance Fields: Local Sampling for Faster Training [0.8223798883838329]
  Recent advances in implicit neural representation from images using neural radiance fields(NeRF) have shown promising results. Some of the limitations of previous NeRF based methods include longer training time, and inaccurate underlying geometry. This paper proposes a depth-guided local sampling strategy and a smaller neural network architecture to achieve faster training time without compromising quality.
  arXiv  Detail & Related papers  (2022-03-26T11:31:35Z)
• T\"oRF: Time-of-Flight Radiance Fields for Dynamic Scene View Synthesis [32.878225196378374]
  We introduce a neural representation based on an image formation model for continuous-wave ToF cameras. We show that this approach improves robustness of dynamic scene reconstruction to erroneous calibration and large motions.
  arXiv  Detail & Related papers  (2021-09-30T17:12:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.