LidaRF: Delving into Lidar for Neural Radiance Field on Street Scenes

• URL: http://arxiv.org/abs/2405.00900v2
• Date: Sat, 4 May 2024 05:36:12 GMT
• Title: LidaRF: Delving into Lidar for Neural Radiance Field on Street Scenes
• Authors: Shanlin Sun, Bingbing Zhuang, Ziyu Jiang, Buyu Liu, Xiaohui Xie, Manmohan Chandraker,
• Abstract summary: Photorealistic simulation plays a crucial role in applications such as autonomous driving. However, reconstruction quality suffers on street scenes due to collinear camera motions and sparser samplings at higher speeds. We propose several insights that allow a better utilization of Lidar data to improve NeRF quality on street scenes.
• Score: 73.65115834242866
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Photorealistic simulation plays a crucial role in applications such as autonomous driving, where advances in neural radiance fields (NeRFs) may allow better scalability through the automatic creation of digital 3D assets. However, reconstruction quality suffers on street scenes due to largely collinear camera motions and sparser samplings at higher speeds. On the other hand, the application often demands rendering from camera views that deviate from the inputs to accurately simulate behaviors like lane changes. In this paper, we propose several insights that allow a better utilization of Lidar data to improve NeRF quality on street scenes. First, our framework learns a geometric scene representation from Lidar, which is fused with the implicit grid-based representation for radiance decoding, thereby supplying stronger geometric information offered by explicit point cloud. Second, we put forth a robust occlusion-aware depth supervision scheme, which allows utilizing densified Lidar points by accumulation. Third, we generate augmented training views from Lidar points for further improvement. Our insights translate to largely improved novel view synthesis under real driving scenes.

Related papers

• Drone-assisted Road Gaussian Splatting with Cross-view Uncertainty [10.37108303188536]
  3D Gaussian Splatting (3D-GS) has made groundbreaking progress in neural rendering. The general fidelity of large-scale road scene renderings is often limited by the input imagery. We introduce the cross-view uncertainty to 3D-GS by matching the car-view ensemble-based rendering uncertainty to aerial images.
  arXiv  Detail & Related papers  (2024-08-27T17:59:55Z)
• SGD: Street View Synthesis with Gaussian Splatting and Diffusion Prior [53.52396082006044]
  Current methods struggle to maintain rendering quality at the viewpoint that deviates significantly from the training viewpoints. This issue stems from the sparse training views captured by a fixed camera on a moving vehicle. We propose a novel approach that enhances the capacity of 3DGS by leveraging prior from a Diffusion Model.
  arXiv  Detail & Related papers  (2024-03-29T09:20:29Z)
• Neural Rendering based Urban Scene Reconstruction for Autonomous Driving [8.007494499012624]
  We propose a multimodal 3D scene reconstruction using a framework combining neural implicit surfaces and radiance fields. Dense 3D reconstruction has many applications in automated driving including automated annotation validation. We demonstrate qualitative and quantitative results on challenging automotive scenes.
  arXiv  Detail & Related papers  (2024-02-09T23:20:23Z)
• Spatiotemporally Consistent HDR Indoor Lighting Estimation [66.26786775252592]
  We propose a physically-motivated deep learning framework to solve the indoor lighting estimation problem. Given a single LDR image with a depth map, our method predicts spatially consistent lighting at any given image position. Our framework achieves photorealistic lighting prediction with higher quality compared to state-of-the-art single-image or video-based methods.
  arXiv  Detail & Related papers  (2023-05-07T20:36:29Z)
• S-NeRF: Neural Radiance Fields for Street Views [23.550385153214688]
  We propose a new street-view NeRF (S-NeRF) that considers novel view synthesis of both the large-scale background scenes and the foreground moving vehicles jointly. Specifically, we improve the scene parameterization function and the camera poses for learning better neural representations from street views. Our method beats the state-of-the-art rivals by reducing 7% to 40% of the mean-squared error in the street-view and a 45% PSNR gain for the moving vehicles rendering.
  arXiv  Detail & Related papers  (2023-03-01T18:59:30Z)
• AligNeRF: High-Fidelity Neural Radiance Fields via Alignment-Aware Training [100.33713282611448]
  We conduct the first pilot study on training NeRF with high-resolution data. We propose the corresponding solutions, including marrying the multilayer perceptron with convolutional layers. Our approach is nearly free without introducing obvious training/testing costs.
  arXiv  Detail & Related papers  (2022-11-17T17:22:28Z)
• CLONeR: Camera-Lidar Fusion for Occupancy Grid-aided Neural Representations [77.90883737693325]
  This paper proposes CLONeR, which significantly improves upon NeRF by allowing it to model large outdoor driving scenes observed from sparse input sensor views. This is achieved by decoupling occupancy and color learning within the NeRF framework into separate Multi-Layer Perceptrons (MLPs) trained using LiDAR and camera data, respectively. In addition, this paper proposes a novel method to build differentiable 3D Occupancy Grid Maps (OGM) alongside the NeRF model, and leverage this occupancy grid for improved sampling of points along a ray for rendering in metric space.
  arXiv  Detail & Related papers  (2022-09-02T17:44:50Z)
• BARF: Bundle-Adjusting Neural Radiance Fields [104.97810696435766]
  We propose Bundle-Adjusting Neural Radiance Fields (BARF) for training NeRF from imperfect camera poses. BARF can effectively optimize the neural scene representations and resolve large camera pose misalignment at the same time. This enables view synthesis and localization of video sequences from unknown camera poses, opening up new avenues for visual localization systems.
  arXiv  Detail & Related papers  (2021-04-13T17:59:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.