Density Uncertainty Quantification with NeRF-Ensembles: Impact of Data and Scene Constraints

• URL: http://arxiv.org/abs/2312.14664v1
• Date: Fri, 22 Dec 2023 13:01:21 GMT
• Title: Density Uncertainty Quantification with NeRF-Ensembles: Impact of Data and Scene Constraints
• Authors: Miriam J\"ager, Steven Landgraf, Boris Jutzi
• Abstract summary: We propose to utilize NeRF-Ensembles that provide a density uncertainty estimate alongside the mean density. We demonstrate that data constraints such as low-quality images and poses lead to a degradation of the training process. NeRF-Ensembles not only provide a tool for quantifying the uncertainty but exhibit two promising advantages: Enhanced robustness and artifact removal.
• Score: 6.905060726100166
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In the fields of computer graphics, computer vision and photogrammetry, Neural Radiance Fields (NeRFs) are a major topic driving current research and development. However, the quality of NeRF-generated 3D scene reconstructions and subsequent surface reconstructions, heavily relies on the network output, particularly the density. Regarding this critical aspect, we propose to utilize NeRF-Ensembles that provide a density uncertainty estimate alongside the mean density. We demonstrate that data constraints such as low-quality images and poses lead to a degradation of the training process, increased density uncertainty and decreased predicted density. Even with high-quality input data, the density uncertainty varies based on scene constraints such as acquisition constellations, occlusions and material properties. NeRF-Ensembles not only provide a tool for quantifying the uncertainty but exhibit two promising advantages: Enhanced robustness and artifact removal. Through the utilization of NeRF-Ensembles instead of single NeRFs, small outliers are removed, yielding a smoother output with improved completeness of structures. Furthermore, applying percentile-based thresholds on density uncertainty outliers proves to be effective for the removal of large (foggy) artifacts in post-processing. We conduct our methodology on 3 different datasets: (i) synthetic benchmark dataset, (ii) real benchmark dataset, (iii) real data under realistic recording conditions and sensors.

Related papers

• Regulating Model Reliance on Non-Robust Features by Smoothing Input Marginal Density [93.32594873253534]
  Trustworthy machine learning requires meticulous regulation of model reliance on non-robust features. We propose a framework to delineate and regulate such features by attributing model predictions to the input.
  arXiv  Detail & Related papers  (2024-07-05T09:16:56Z)
• Bayesian NeRF: Quantifying Uncertainty with Volume Density for Neural Implicit Fields [1.199955563466263]
  We present a Bayesian Neural Radiance Field (NeRF), which explicitly quantifies uncertainty in the volume density by modeling uncertainty in the occupancy. NeRF diverges from traditional geometric methods by providing an enriched scene representation, rendering color and density in 3D space from various viewpoints. We show that our method significantly enhances performance on RGB and depth images in a comprehensive dataset.
  arXiv  Detail & Related papers  (2024-04-10T04:24:42Z)
• FILP-3D: Enhancing 3D Few-shot Class-incremental Learning with Pre-trained Vision-Language Models [59.13757801286343]
  Few-shot class-incremental learning aims to mitigate the catastrophic forgetting issue when a model is incrementally trained on limited data. We introduce the FILP-3D framework with two novel components: the Redundant Feature Eliminator (RFE) for feature space misalignment and the Spatial Noise Compensator (SNC) for significant noise.
  arXiv  Detail & Related papers  (2023-12-28T14:52:07Z)
• Leveraging Neural Radiance Fields for Uncertainty-Aware Visual Localization [56.95046107046027]
  We propose to leverage Neural Radiance Fields (NeRF) to generate training samples for scene coordinate regression. Despite NeRF's efficiency in rendering, many of the rendered data are polluted by artifacts or only contain minimal information gain.
  arXiv  Detail & Related papers  (2023-10-10T20:11:13Z)
• OOD-CV-v2: An extended Benchmark for Robustness to Out-of-Distribution Shifts of Individual Nuisances in Natural Images [59.51657161097337]
  OOD-CV-v2 is a benchmark dataset that includes out-of-distribution examples of 10 object categories in terms of pose, shape, texture, context and the weather conditions. In addition to this novel dataset, we contribute extensive experiments using popular baseline methods.
  arXiv  Detail & Related papers  (2023-04-17T20:39:25Z)
• On the Importance of Accurate Geometry Data for Dense 3D Vision Tasks [61.74608497496841]
  Training on inaccurate or corrupt data induces model bias and hampers generalisation capabilities. This paper investigates the effect of sensor errors for the dense 3D vision tasks of depth estimation and reconstruction.
  arXiv  Detail & Related papers  (2023-03-26T22:32:44Z)
• Density-aware NeRF Ensembles: Quantifying Predictive Uncertainty in Neural Radiance Fields [7.380217868660371]
  We show that ensembling effectively quantifies model uncertainty in Neural Radiance Fields (NeRFs) We demonstrate that NeRF uncertainty can be utilised for next-best view selection and model refinement.
  arXiv  Detail & Related papers  (2022-09-19T02:28:33Z)
• Pre-training via Denoising for Molecular Property Prediction [53.409242538744444]
  We describe a pre-training technique that utilizes large datasets of 3D molecular structures at equilibrium. Inspired by recent advances in noise regularization, our pre-training objective is based on denoising.
  arXiv  Detail & Related papers  (2022-05-31T22:28:34Z)
• Physics-Driven Learning of Wasserstein GAN for Density Reconstruction in Dynamic Tomography [4.970364068620608]
  In this work, we demonstrate the ability of learned deep neural networks to perform artifact removal in noisy density reconstructions. We use a Wasserstein generative adversarial network (WGAN), where the generator serves as a denoiser that removes artifacts in densities obtained from traditional reconstruction algorithms. Preliminary numerical results show that the models trained in our frameworks can remove significant portions of unknown noise in density time-series data.
  arXiv  Detail & Related papers  (2021-10-28T20:23:06Z)
• A Unified Plug-and-Play Framework for Effective Data Denoising and Robust Abstention [4.200576272300216]
  We propose a unified filtering framework leveraging underlying data density. Our framework can effectively denoising training data and avoid predicting uncertain test data points.
  arXiv  Detail & Related papers  (2020-09-25T04:18:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.