Uncertainty-Driven Active Vision for Implicit Scene Reconstruction

• URL: http://arxiv.org/abs/2210.00978v1
• Date: Mon, 3 Oct 2022 14:45:54 GMT
• Title: Uncertainty-Driven Active Vision for Implicit Scene Reconstruction
• Authors: Edward J. Smith and Michal Drozdzal and Derek Nowrouzezahrai and David Meger and Adriana Romero-Soriano
• Abstract summary: We propose an uncertainty-driven active vision approach for implicit scene reconstruction. We develop an occupancy-based reconstruction method which accurately represents scenes using either 2D or 3D supervision.
• Score: 31.890470553111122
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Multi-view implicit scene reconstruction methods have become increasingly popular due to their ability to represent complex scene details. Recent efforts have been devoted to improving the representation of input information and to reducing the number of views required to obtain high quality reconstructions. Yet, perhaps surprisingly, the study of which views to select to maximally improve scene understanding remains largely unexplored. We propose an uncertainty-driven active vision approach for implicit scene reconstruction, which leverages occupancy uncertainty accumulated across the scene using volume rendering to select the next view to acquire. To this end, we develop an occupancy-based reconstruction method which accurately represents scenes using either 2D or 3D supervision. We evaluate our proposed approach on the ABC dataset and the in the wild CO3D dataset, and show that: (1) we are able to obtain high quality state-of-the-art occupancy reconstructions; (2) our perspective conditioned uncertainty definition is effective to drive improvements in next best view selection and outperforms strong baseline approaches; and (3) we can further improve shape understanding by performing a gradient-based search on the view selection candidates. Overall, our results highlight the importance of view selection for implicit scene reconstruction, making it a promising avenue to explore further.

Related papers

• Frequency-based View Selection in Gaussian Splatting Reconstruction [9.603843571051744]
  We investigate the problem of active view selection to perform 3D Gaussian Splatting reconstructions with as few input images as possible. By ranking the potential views in the frequency domain, we are able to effectively estimate the potential information gain of new viewpoints. Our method achieves state-of-the-art results in view selection, demonstrating its potential for efficient image-based 3D reconstruction.
  arXiv  Detail & Related papers  (2024-09-24T21:44:26Z)
• PVP-Recon: Progressive View Planning via Warping Consistency for Sparse-View Surface Reconstruction [49.7580491592023]
  We propose PVP-Recon, a novel and effective sparse-view surface reconstruction method. PVP-Recon starts initial surface reconstruction with as few as 3 views and progressively adds new views. This progressive view planning progress is interleaved with a neural SDF-based reconstruction module.
  arXiv  Detail & Related papers  (2024-09-09T10:06:34Z)
• MaRINeR: Enhancing Novel Views by Matching Rendered Images with Nearby References [49.71130133080821]
  MaRINeR is a refinement method that leverages information of a nearby mapping image to improve the rendering of a target viewpoint. We show improved renderings in quantitative metrics and qualitative examples from both explicit and implicit scene representations.
  arXiv  Detail & Related papers  (2024-07-18T17:50:03Z)
• Semantic-aware Next-Best-View for Multi-DoFs Mobile System in Search-and-Acquisition based Visual Perception [20.480581428768854]
  We formulate a novel information gain that integrates both visibility gain and semantic gain in a unified form to select the semantic-aware Next-Best-View. Several semantically relevant reconstruction metrics, including perspective directivity and region of interest (ROI)-to-full reconstruction volume ratio, are introduced to evaluate the performance of the proposed approach.
  arXiv  Detail & Related papers  (2024-04-25T11:01:40Z)
• Autonomous Implicit Indoor Scene Reconstruction with Frontier Exploration [10.975244524831696]
  Implicit neural representations have demonstrated significant promise for 3D scene reconstruction. Recent works have extended their applications to autonomous implicit reconstruction through the Next Best View (NBV) based method. We propose to incorporate frontier-based exploration tasks for global coverage with implicit surface uncertainty-based reconstruction tasks.
  arXiv  Detail & Related papers  (2024-04-16T01:59:03Z)
• Reusable Architecture Growth for Continual Stereo Matching [92.36221737921274]
  We introduce a Reusable Architecture Growth (RAG) framework to learn new scenes continually in both supervised and self-supervised manners. RAG can maintain high reusability during growth by reusing previous units while obtaining good performance. We also present a Scene Router module to adaptively select the scene-specific architecture path at inference.
  arXiv  Detail & Related papers  (2024-03-30T13:24:58Z)
• RadOcc: Learning Cross-Modality Occupancy Knowledge through Rendering Assisted Distillation [50.35403070279804]
  3D occupancy prediction is an emerging task that aims to estimate the occupancy states and semantics of 3D scenes using multi-view images. We propose RadOcc, a Rendering assisted distillation paradigm for 3D Occupancy prediction.
  arXiv  Detail & Related papers  (2023-12-19T03:39:56Z)
• Neural Radiance Field Codebooks [53.01356339021285]
  We introduce Neural Radiance Field Codebooks (NRC), a scalable method for learning object-centric representations. NRC learns to reconstruct scenes from novel views using a dictionary of object codes which are decoded through a volumetric reconstruction. We show that NRC representations transfer well to object navigation in THOR, outperforming 2D and 3D representation learning methods by 3.1% success rate.
  arXiv  Detail & Related papers  (2023-01-10T18:03:48Z)
• NeurAR: Neural Uncertainty for Autonomous 3D Reconstruction [64.36535692191343]
  Implicit neural representations have shown compelling results in offline 3D reconstruction and also recently demonstrated the potential for online SLAM systems. This paper addresses two key challenges: 1) seeking a criterion to measure the quality of the candidate viewpoints for the view planning based on the new representations, and 2) learning the criterion from data that can generalize to different scenes instead of hand-crafting one. Our method demonstrates significant improvements on various metrics for the rendered image quality and the geometry quality of the reconstructed 3D models when compared with variants using TSDF or reconstruction without view planning.
  arXiv  Detail & Related papers  (2022-07-22T10:05:36Z)
• 3D-NVS: A 3D Supervision Approach for Next View Selection [22.662440687566587]
  We present a classification based approach for the next best view selection. We show how we can plausibly obtain a supervisory signal for this task.
  arXiv  Detail & Related papers  (2020-12-03T07:50:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.