Leveraging Neural Radiance Fields for Uncertainty-Aware Visual Localization

• URL: http://arxiv.org/abs/2310.06984v1
• Date: Tue, 10 Oct 2023 20:11:13 GMT
• Title: Leveraging Neural Radiance Fields for Uncertainty-Aware Visual Localization
• Authors: Le Chen, Weirong Chen, Rui Wang, Marc Pollefeys
• Abstract summary: We propose to leverage Neural Radiance Fields (NeRF) to generate training samples for scene coordinate regression. Despite NeRF's efficiency in rendering, many of the rendered data are polluted by artifacts or only contain minimal information gain.
• Score: 56.95046107046027
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As a promising fashion for visual localization, scene coordinate regression (SCR) has seen tremendous progress in the past decade. Most recent methods usually adopt neural networks to learn the mapping from image pixels to 3D scene coordinates, which requires a vast amount of annotated training data. We propose to leverage Neural Radiance Fields (NeRF) to generate training samples for SCR. Despite NeRF's efficiency in rendering, many of the rendered data are polluted by artifacts or only contain minimal information gain, which can hinder the regression accuracy or bring unnecessary computational costs with redundant data. These challenges are addressed in three folds in this paper: (1) A NeRF is designed to separately predict uncertainties for the rendered color and depth images, which reveal data reliability at the pixel level. (2) SCR is formulated as deep evidential learning with epistemic uncertainty, which is used to evaluate information gain and scene coordinate quality. (3) Based on the three arts of uncertainties, a novel view selection policy is formed that significantly improves data efficiency. Experiments on public datasets demonstrate that our method could select the samples that bring the most information gain and promote the performance with the highest efficiency.

Related papers

• SG-NeRF: Neural Surface Reconstruction with Scene Graph Optimization [16.460851701725392]
  We present a novel approach that optimize radiance fields with scene graphs to mitigate the influence of outlier poses. Our method incorporates an adaptive inlier-outlier confidence estimation scheme based on scene graphs. We also introduce an effective intersection-over-union (IoU) loss to optimize the camera pose and surface geometry.
  arXiv  Detail & Related papers  (2024-07-17T15:50:17Z)
• Leveraging Neural Radiance Field in Descriptor Synthesis for Keypoints Scene Coordinate Regression [1.2974519529978974]
  This paper introduces a pipeline for keypoint descriptor synthesis using Neural Radiance Field (NeRF) generating novel poses and feeding them into a trained NeRF model to create new views, our approach enhances the KSCR's capabilities in data-scarce environments. The proposed system could significantly improve localization accuracy by up to 50% and cost only a fraction of time for data synthesis.
  arXiv  Detail & Related papers  (2024-03-15T13:40:37Z)
• Ray Denoising: Depth-aware Hard Negative Sampling for Multi-view 3D Object Detection [46.041193889845474]
  Ray Denoising is an innovative method that enhances detection accuracy by strategically sampling along camera rays to construct hard negative examples. Ray Denoising is designed as a plug-and-play module, compatible with any DETR-style multi-view 3D detectors. It achieves a 1.9% improvement in mean Average Precision (mAP) over the state-of-the-art StreamPETR method on the NuScenes dataset.
  arXiv  Detail & Related papers  (2024-02-06T02:17:44Z)
• S3IM: Stochastic Structural SIMilarity and Its Unreasonable Effectiveness for Neural Fields [46.9880016170926]
  We show that Structural SIMilarity (S3IM) loss processes multiple data points as a whole set instead of multiplexing multiple inputs independently. Our experiments demonstrate the unreasonable effectiveness of S3IM in improving NeRF and neural surface representation for nearly free.
  arXiv  Detail & Related papers  (2023-08-14T09:45:28Z)
• Neural Point-based Volumetric Avatar: Surface-guided Neural Points for Efficient and Photorealistic Volumetric Head Avatar [62.87222308616711]
  We propose fullname (name), a method that adopts the neural point representation and the neural volume rendering process. Specifically, the neural points are strategically constrained around the surface of the target expression via a high-resolution UV displacement map. By design, our name is better equipped to handle topologically changing regions and thin structures while also ensuring accurate expression control when animating avatars.
  arXiv  Detail & Related papers  (2023-07-11T03:40:10Z)
• AligNeRF: High-Fidelity Neural Radiance Fields via Alignment-Aware Training [100.33713282611448]
  We conduct the first pilot study on training NeRF with high-resolution data. We propose the corresponding solutions, including marrying the multilayer perceptron with convolutional layers. Our approach is nearly free without introducing obvious training/testing costs.
  arXiv  Detail & Related papers  (2022-11-17T17:22:28Z)
• CLONeR: Camera-Lidar Fusion for Occupancy Grid-aided Neural Representations [77.90883737693325]
  This paper proposes CLONeR, which significantly improves upon NeRF by allowing it to model large outdoor driving scenes observed from sparse input sensor views. This is achieved by decoupling occupancy and color learning within the NeRF framework into separate Multi-Layer Perceptrons (MLPs) trained using LiDAR and camera data, respectively. In addition, this paper proposes a novel method to build differentiable 3D Occupancy Grid Maps (OGM) alongside the NeRF model, and leverage this occupancy grid for improved sampling of points along a ray for rendering in metric space.
  arXiv  Detail & Related papers  (2022-09-02T17:44:50Z)
• Learning Collision-Free Space Detection from Stereo Images: Homography Matrix Brings Better Data Augmentation [16.99302954185652]
  It remains an open challenge to train deep convolutional neural networks (DCNNs) using only a small quantity of training samples. This paper explores an effective training data augmentation approach that can be employed to improve the overall DCNN performance.
  arXiv  Detail & Related papers  (2020-12-14T19:14:35Z)
• Probabilistic 3D surface reconstruction from sparse MRI information [58.14653650521129]
  We present a novel probabilistic deep learning approach for concurrent 3D surface reconstruction from sparse 2D MR image data and aleatoric uncertainty prediction. Our method is capable of reconstructing large surface meshes from three quasi-orthogonal MR imaging slices from limited training sets.
  arXiv  Detail & Related papers  (2020-10-05T14:18:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.