Finding Waldo: Towards Efficient Exploration of NeRF Scene Spaces

• URL: http://arxiv.org/abs/2403.04508v2
• Date: Fri, 8 Mar 2024 10:34:48 GMT
• Title: Finding Waldo: Towards Efficient Exploration of NeRF Scene Spaces
• Authors: Evangelos Skartados, Mehmet Kerim Yucel, Bruno Manganelli, Anastasios Drosou, Albert Sa\`a-Garriga
• Abstract summary: We propose and formally define the scene exploration framework as the efficient discovery of NeRF model inputs. To remedy the lack of approaches addressing scene exploration, we first propose two baseline methods called Guided-Random Search (GRS) and Pose Interpolation-based Search (PIBS) We then cast scene exploration as an optimization problem, and propose the criteria-agnostic Evolution-Guided Pose Search (EGPS) for exploration.
• Score: 4.944459754818577
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural Radiance Fields (NeRF) have quickly become the primary approach for 3D reconstruction and novel view synthesis in recent years due to their remarkable performance. Despite the huge interest in NeRF methods, a practical use case of NeRFs has largely been ignored; the exploration of the scene space modelled by a NeRF. In this paper, for the first time in the literature, we propose and formally define the scene exploration framework as the efficient discovery of NeRF model inputs (i.e. coordinates and viewing angles), using which one can render novel views that adhere to user-selected criteria. To remedy the lack of approaches addressing scene exploration, we first propose two baseline methods called Guided-Random Search (GRS) and Pose Interpolation-based Search (PIBS). We then cast scene exploration as an optimization problem, and propose the criteria-agnostic Evolution-Guided Pose Search (EGPS) for efficient exploration. We test all three approaches with various criteria (e.g. saliency maximization, image quality maximization, photo-composition quality improvement) and show that our EGPS performs more favourably than other baselines. We finally highlight key points and limitations, and outline directions for future research in scene exploration.

Related papers

• FrontierNet: Learning Visual Cues to Explore [54.8265603996238]
  This work aims at leveraging 2D visual cues for efficient autonomous exploration, addressing the limitations of extracting goal poses from a 3D map. We propose a image-only frontier-based exploration system, with FrontierNet as a core component developed in this work. Our approach provides an alternative to existing 3D-dependent exploration systems, achieving a 16% improvement in early-stage exploration efficiency.
  arXiv  Detail & Related papers  (2025-01-08T16:25:32Z)
• SG-NeRF: Neural Surface Reconstruction with Scene Graph Optimization [16.460851701725392]
  We present a novel approach that optimize radiance fields with scene graphs to mitigate the influence of outlier poses. Our method incorporates an adaptive inlier-outlier confidence estimation scheme based on scene graphs. We also introduce an effective intersection-over-union (IoU) loss to optimize the camera pose and surface geometry.
  arXiv  Detail & Related papers  (2024-07-17T15:50:17Z)
• ProvNeRF: Modeling per Point Provenance in NeRFs as a Stochastic Field [52.09661042881063]
  We propose an approach that models the bfprovenance for each point -- i.e., the locations where it is likely visible -- of NeRFs as a text field. We show that modeling per-point provenance during the NeRF optimization enriches the model with information on leading to improvements in novel view synthesis and uncertainty estimation.
  arXiv  Detail & Related papers  (2024-01-16T06:19:18Z)
• NeRF-Enhanced Outpainting for Faithful Field-of-View Extrapolation [18.682430719467202]
  In various applications, such as robotic navigation and remote visual assistance, expanding the field of view (FOV) of the camera proves beneficial for enhancing environmental perception. We formulate a new problem of faithful FOV extrapolation that utilizes a set of pre-captured images as prior knowledge of the scene. We present NeRF-Enhanced Outpainting (NEO) that uses extended-FOV images generated through NeRF to train a scene-specific image outpainting model.
  arXiv  Detail & Related papers  (2023-09-23T03:16:58Z)
• SCADE: NeRFs from Space Carving with Ambiguity-Aware Depth Estimates [16.344734292989504]
  SCADE is a novel technique that improves NeRF reconstruction quality on sparse, unconstrained input views. We propose a new method that learns to predict, for each view, a continuous, multimodal distribution of depth estimates. Experiments show that our approach enables higher fidelity novel view synthesis from sparse views.
  arXiv  Detail & Related papers  (2023-03-23T18:00:07Z)
• DARF: Depth-Aware Generalizable Neural Radiance Field [51.29437249009986]
  We propose the Depth-Aware Generalizable Neural Radiance Field (DARF) with a Depth-Aware Dynamic Sampling (DADS) strategy. Our framework infers the unseen scenes on both pixel level and geometry level with only a few input images. Compared with state-of-the-art generalizable NeRF methods, DARF reduces samples by 50%, while improving rendering quality and depth estimation.
  arXiv  Detail & Related papers  (2022-12-05T14:00:59Z)
• Searching a High-Performance Feature Extractor for Text Recognition Network [92.12492627169108]
  We design a domain-specific search space by exploring principles for having good feature extractors. As the space is huge and complexly structured, no existing NAS algorithms can be applied. We propose a two-stage algorithm to effectively search in the space.
  arXiv  Detail & Related papers  (2022-09-27T03:49:04Z)
• CLONeR: Camera-Lidar Fusion for Occupancy Grid-aided Neural Representations [77.90883737693325]
  This paper proposes CLONeR, which significantly improves upon NeRF by allowing it to model large outdoor driving scenes observed from sparse input sensor views. This is achieved by decoupling occupancy and color learning within the NeRF framework into separate Multi-Layer Perceptrons (MLPs) trained using LiDAR and camera data, respectively. In addition, this paper proposes a novel method to build differentiable 3D Occupancy Grid Maps (OGM) alongside the NeRF model, and leverage this occupancy grid for improved sampling of points along a ray for rendering in metric space.
  arXiv  Detail & Related papers  (2022-09-02T17:44:50Z)
• GNeRF: GAN-based Neural Radiance Field without Posed Camera [67.80805274569354]
  We introduce GNeRF, a framework to marry Generative Adversarial Networks (GAN) with Neural Radiance Field reconstruction for the complex scenarios with unknown and even randomly camera poses. Our approach outperforms the baselines favorably in those scenes with repeated patterns or even low textures that are regarded as extremely challenging before.
  arXiv  Detail & Related papers  (2021-03-29T13:36:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.