Neural Surface Reconstruction and Rendering for LiDAR-Visual Systems

• URL: http://arxiv.org/abs/2409.05310v1
• Date: Mon, 9 Sep 2024 03:42:12 GMT
• Title: Neural Surface Reconstruction and Rendering for LiDAR-Visual Systems
• Authors: Jianheng Liu, Chunran Zheng, Yunfei Wan, Bowen Wang, Yixi Cai, Fu Zhang,
• Abstract summary: This paper presents a unified surface reconstruction and rendering framework for LiDAR-visual systems. We address the structural visible gap between NeRF and NDF by utilizing a visible-aware occupancy map. We unify the training of the NDF and NeRF using a spatial-varying scale SDF-to-density transformation.
• Score: 9.325214913221252
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper presents a unified surface reconstruction and rendering framework for LiDAR-visual systems, integrating Neural Radiance Fields (NeRF) and Neural Distance Fields (NDF) to recover both appearance and structural information from posed images and point clouds. We address the structural visible gap between NeRF and NDF by utilizing a visible-aware occupancy map to classify space into the free, occupied, visible unknown, and background regions. This classification facilitates the recovery of a complete appearance and structure of the scene. We unify the training of the NDF and NeRF using a spatial-varying scale SDF-to-density transformation for levels of detail for both structure and appearance. The proposed method leverages the learned NDF for structure-aware NeRF training by an adaptive sphere tracing sampling strategy for accurate structure rendering. In return, NeRF further refines structural in recovering missing or fuzzy structures in the NDF. Extensive experiments demonstrate the superior quality and versatility of the proposed method across various scenarios. To benefit the community, the codes will be released at \url{https://github.com/hku-mars/M2Mapping}.

Related papers

• NeuRodin: A Two-stage Framework for High-Fidelity Neural Surface Reconstruction [63.85586195085141]
  Signed Distance Function (SDF)-based volume rendering has demonstrated significant capabilities in surface reconstruction. We introduce NeuRodin, a novel two-stage neural surface reconstruction framework. NeuRodin achieves high-fidelity surface reconstruction and retains the flexible optimization characteristics of density-based methods.
  arXiv  Detail & Related papers  (2024-08-19T17:36:35Z)
• NC-SDF: Enhancing Indoor Scene Reconstruction Using Neural SDFs with View-Dependent Normal Compensation [13.465401006826294]
  We present NC-SDF, a neural signed distance field (SDF) 3D reconstruction framework with view-dependent normal compensation (NC) By adaptively learning and correcting the biases, our NC-SDF effectively mitigates the adverse impact of inconsistent supervision. Experiments on synthetic and real-world datasets demonstrate that NC-SDF outperforms existing approaches in terms of reconstruction quality.
  arXiv  Detail & Related papers  (2024-05-01T06:26:35Z)
• The NeRFect Match: Exploring NeRF Features for Visual Localization [45.42725089658268]
  We propose the use of Neural Radiance Fields (NeRF) as a scene representation for visual localization. We extend its recognized advantages by exploring the potential of NeRF's internal features in establishing precise 2D-3D matches for localization. We introduce NeRFMatch, an advanced 2D-3D matching function that capitalizes on the internal knowledge of NeRF learned via view synthesis.
  arXiv  Detail & Related papers  (2024-03-14T17:11:49Z)
• PNeRFLoc: Visual Localization with Point-based Neural Radiance Fields [54.8553158441296]
  We propose a novel visual localization framework, ie, PNeRFLoc, based on a unified point-based representation. On the one hand, PNeRFLoc supports the initial pose estimation by matching 2D and 3D feature points. On the other hand, it also enables pose refinement with novel view synthesis using rendering-based optimization.
  arXiv  Detail & Related papers  (2023-12-17T08:30:00Z)
• LiDAR-NeRF: Novel LiDAR View Synthesis via Neural Radiance Fields [112.62936571539232]
  We introduce a new task, novel view synthesis for LiDAR sensors. Traditional model-based LiDAR simulators with style-transfer neural networks can be applied to render novel views. We use a neural radiance field (NeRF) to facilitate the joint learning of geometry and the attributes of 3D points.
  arXiv  Detail & Related papers  (2023-04-20T15:44:37Z)
• NeRFVS: Neural Radiance Fields for Free View Synthesis via Geometry Scaffolds [60.1382112938132]
  We present NeRFVS, a novel neural radiance fields (NeRF) based method to enable free navigation in a room. NeRF achieves impressive performance in rendering images for novel views similar to the input views while suffering for novel views that are significantly different from the training views.
  arXiv  Detail & Related papers  (2023-04-13T06:40:08Z)
• NeuralUDF: Learning Unsigned Distance Fields for Multi-view Reconstruction of Surfaces with Arbitrary Topologies [87.06532943371575]
  We present a novel method, called NeuralUDF, for reconstructing surfaces with arbitrary topologies from 2D images via volume rendering. In this paper, we propose to represent surfaces as the Unsigned Distance Function (UDF) and develop a new volume rendering scheme to learn the neural UDF representation.
  arXiv  Detail & Related papers  (2022-11-25T15:21:45Z)
• StructNeRF: Neural Radiance Fields for Indoor Scenes with Structural Hints [23.15914545835831]
  StructNeRF is a solution to novel view synthesis for indoor scenes with sparse inputs. Our method improves both the geometry and the view synthesis performance of NeRF without any additional training on external data.
  arXiv  Detail & Related papers  (2022-09-12T14:33:27Z)
• CLONeR: Camera-Lidar Fusion for Occupancy Grid-aided Neural Representations [77.90883737693325]
  This paper proposes CLONeR, which significantly improves upon NeRF by allowing it to model large outdoor driving scenes observed from sparse input sensor views. This is achieved by decoupling occupancy and color learning within the NeRF framework into separate Multi-Layer Perceptrons (MLPs) trained using LiDAR and camera data, respectively. In addition, this paper proposes a novel method to build differentiable 3D Occupancy Grid Maps (OGM) alongside the NeRF model, and leverage this occupancy grid for improved sampling of points along a ray for rendering in metric space.
  arXiv  Detail & Related papers  (2022-09-02T17:44:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.