NeRF-LiDAR: Generating Realistic LiDAR Point Clouds with Neural Radiance Fields

• URL: http://arxiv.org/abs/2304.14811v3
• Date: Sat, 20 Jan 2024 12:56:59 GMT
• Title: NeRF-LiDAR: Generating Realistic LiDAR Point Clouds with Neural Radiance Fields
• Authors: Junge Zhang, Feihu Zhang, Shaochen Kuang, Li Zhang
• Abstract summary: We present NeRF-LIDAR, a novel LiDAR simulation method that leverages real-world information to generate realistic LIDAR point clouds. We verify the effectiveness of our NeRF-LiDAR by training different 3D segmentation models on the generated LiDAR point clouds.
• Score: 20.887421720818892
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Labeling LiDAR point clouds for training autonomous driving is extremely expensive and difficult. LiDAR simulation aims at generating realistic LiDAR data with labels for training and verifying self-driving algorithms more efficiently. Recently, Neural Radiance Fields (NeRF) have been proposed for novel view synthesis using implicit reconstruction of 3D scenes. Inspired by this, we present NeRF-LIDAR, a novel LiDAR simulation method that leverages real-world information to generate realistic LIDAR point clouds. Different from existing LiDAR simulators, we use real images and point cloud data collected by self-driving cars to learn the 3D scene representation, point cloud generation and label rendering. We verify the effectiveness of our NeRF-LiDAR by training different 3D segmentation models on the generated LiDAR point clouds. It reveals that the trained models are able to achieve similar accuracy when compared with the same model trained on the real LiDAR data. Besides, the generated data is capable of boosting the accuracy through pre-training which helps reduce the requirements of the real labeled data.

Related papers

• LiDAR-GS:Real-time LiDAR Re-Simulation using Gaussian Splatting [50.808933338389686]
  LiDAR simulation plays a crucial role in closed-loop simulation for autonomous driving. We present LiDAR-GS, the first LiDAR Gaussian Splatting method, for real-time high-fidelity re-simulation of LiDAR sensor scans in public urban road scenes. Our approach succeeds in simultaneously re-simulating depth, intensity, and ray-drop channels, achieving state-of-the-art results in both rendering frame rate and quality on publically available large scene datasets.
  arXiv  Detail & Related papers  (2024-10-07T15:07:56Z)
• UltraLiDAR: Learning Compact Representations for LiDAR Completion and Generation [51.443788294845845]
  We present UltraLiDAR, a data-driven framework for scene-level LiDAR completion, LiDAR generation, and LiDAR manipulation. We show that by aligning the representation of a sparse point cloud to that of a dense point cloud, we can densify the sparse point clouds. By learning a prior over the discrete codebook, we can generate diverse, realistic LiDAR point clouds for self-driving.
  arXiv  Detail & Related papers  (2023-11-02T17:57:03Z)
• LiDAR Data Synthesis with Denoising Diffusion Probabilistic Models [1.1965844936801797]
  Generative modeling of 3D LiDAR data is an emerging task with promising applications for autonomous mobile robots. We present R2DM, a novel generative model for LiDAR data that can generate diverse and high-fidelity 3D scene point clouds. Our method is built upon denoising diffusion probabilistic models (DDPMs), which have shown impressive results among generative model frameworks.
  arXiv  Detail & Related papers  (2023-09-17T12:26:57Z)
• LiDAR View Synthesis for Robust Vehicle Navigation Without Expert Labels [50.40632021583213]
  We propose synthesizing additional LiDAR point clouds from novel viewpoints without physically driving at dangerous positions. We train a deep learning model, which takes a LiDAR scan as input and predicts the future trajectory as output. A waypoint controller is then applied to this predicted trajectory to determine the throttle and steering labels of the ego-vehicle.
  arXiv  Detail & Related papers  (2023-08-02T20:46:43Z)
• LiDAR-NeRF: Novel LiDAR View Synthesis via Neural Radiance Fields [112.62936571539232]
  We introduce a new task, novel view synthesis for LiDAR sensors. Traditional model-based LiDAR simulators with style-transfer neural networks can be applied to render novel views. We use a neural radiance field (NeRF) to facilitate the joint learning of geometry and the attributes of 3D points.
  arXiv  Detail & Related papers  (2023-04-20T15:44:37Z)
• Learning to Simulate Realistic LiDARs [66.7519667383175]
  We introduce a pipeline for data-driven simulation of a realistic LiDAR sensor. We show that our model can learn to encode realistic effects such as dropped points on transparent surfaces. We use our technique to learn models of two distinct LiDAR sensors and use them to improve simulated LiDAR data accordingly.
  arXiv  Detail & Related papers  (2022-09-22T13:12:54Z)
• LiDARsim: Realistic LiDAR Simulation by Leveraging the Real World [84.57894492587053]
  We develop a novel simulator that captures both the power of physics-based and learning-based simulation. We first utilize ray casting over the 3D scene and then use a deep neural network to produce deviations from the physics-based simulation. We showcase LiDARsim's usefulness for perception algorithms-testing on long-tail events and end-to-end closed-loop evaluation on safety-critical scenarios.
  arXiv  Detail & Related papers  (2020-06-16T17:44:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.