Learning to Simulate Realistic LiDARs

• URL: http://arxiv.org/abs/2209.10986v1
• Date: Thu, 22 Sep 2022 13:12:54 GMT
• Title: Learning to Simulate Realistic LiDARs
• Authors: Benoit Guillard, Sai Vemprala, Jayesh K. Gupta, Ondrej Miksik, Vibhav Vineet, Pascal Fua, Ashish Kapoor
• Abstract summary: We introduce a pipeline for data-driven simulation of a realistic LiDAR sensor. We show that our model can learn to encode realistic effects such as dropped points on transparent surfaces. We use our technique to learn models of two distinct LiDAR sensors and use them to improve simulated LiDAR data accordingly.
• Score: 66.7519667383175
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Simulating realistic sensors is a challenging part in data generation for autonomous systems, often involving carefully handcrafted sensor design, scene properties, and physics modeling. To alleviate this, we introduce a pipeline for data-driven simulation of a realistic LiDAR sensor. We propose a model that learns a mapping between RGB images and corresponding LiDAR features such as raydrop or per-point intensities directly from real datasets. We show that our model can learn to encode realistic effects such as dropped points on transparent surfaces or high intensity returns on reflective materials. When applied to naively raycasted point clouds provided by off-the-shelf simulator software, our model enhances the data by predicting intensities and removing points based on the scene's appearance to match a real LiDAR sensor. We use our technique to learn models of two distinct LiDAR sensors and use them to improve simulated LiDAR data accordingly. Through a sample task of vehicle segmentation, we show that enhancing simulated point clouds with our technique improves downstream task performance.

Related papers

• LiDAR-GS:Real-time LiDAR Re-Simulation using Gaussian Splatting [50.808933338389686]
  LiDAR simulation plays a crucial role in closed-loop simulation for autonomous driving. We present LiDAR-GS, the first LiDAR Gaussian Splatting method, for real-time high-fidelity re-simulation of LiDAR sensor scans in public urban road scenes. Our approach succeeds in simultaneously re-simulating depth, intensity, and ray-drop channels, achieving state-of-the-art results in both rendering frame rate and quality on publically available large scene datasets.
  arXiv  Detail & Related papers  (2024-10-07T15:07:56Z)
• RaSim: A Range-aware High-fidelity RGB-D Data Simulation Pipeline for Real-world Applications [55.24463002889]
  We focus on depth data synthesis and develop a range-aware RGB-D data simulation pipeline (RaSim) In particular, high-fidelity depth data is generated by imitating the imaging principle of real-world sensors. RaSim can be directly applied to real-world scenarios without any finetuning and excel at downstream RGB-D perception tasks.
  arXiv  Detail & Related papers  (2024-04-05T08:52:32Z)
• Realistic Rainy Weather Simulation for LiDARs in CARLA Simulator [17.605152926277707]
  We propose a simulator-based physical modeling approach to augment LiDAR data in rainy weather. We complete the modeling task of the rainy weather in the CARLA simulator and establish a pipeline for LiDAR data collection. In the experiment, we observe that the model augmented by the synthetic data improves the object detection task's performance.
  arXiv  Detail & Related papers  (2023-12-20T05:16:04Z)
• GAN-Based LiDAR Intensity Simulation [3.8697834534260447]
  We train GANs to translate between camera images and LiDAR scans from real test drives. We test the performance of the LiDAR simulation by testing how well an object detection network generalizes between real and synthetic point clouds.
  arXiv  Detail & Related papers  (2023-11-26T20:44:09Z)
• Neural LiDAR Fields for Novel View Synthesis [80.45307792404685]
  We present Neural Fields for LiDAR (NFL), a method to optimise a neural field scene representation from LiDAR measurements. NFL combines the rendering power of neural fields with a detailed, physically motivated model of the LiDAR sensing process. We show that the improved realism of the synthesized views narrows the domain gap to real scans and translates to better registration and semantic segmentation performance.
  arXiv  Detail & Related papers  (2023-05-02T17:55:38Z)
• NeRF-LiDAR: Generating Realistic LiDAR Point Clouds with Neural Radiance Fields [20.887421720818892]
  We present NeRF-LIDAR, a novel LiDAR simulation method that leverages real-world information to generate realistic LIDAR point clouds. We verify the effectiveness of our NeRF-LiDAR by training different 3D segmentation models on the generated LiDAR point clouds.
  arXiv  Detail & Related papers  (2023-04-28T12:41:28Z)
• PCGen: Point Cloud Generator for LiDAR Simulation [10.692184635629792]
  Existing methods generate data which are more noisy and complete than the real point clouds. We propose FPA raycasting and surrogate model raydrop. With minimal training data, the surrogate model can generalize to different geographies and scenes. Results show that object detection models trained by simulation data can achieve similar result as the real data trained model.
  arXiv  Detail & Related papers  (2022-10-17T04:13:21Z)
• A Realism Metric for Generated LiDAR Point Clouds [2.6205925938720833]
  This paper presents a novel metric to quantify the realism of LiDAR point clouds. Relevant features are learned from real-world and synthetic point clouds by training on a proxy classification task. In a series of experiments, we demonstrate the application of our metric to determine the realism of generated LiDAR data and compare the realism estimation of our metric to the performance of a segmentation model.
  arXiv  Detail & Related papers  (2022-08-31T16:37:57Z)
• Recovering and Simulating Pedestrians in the Wild [81.38135735146015]
  We propose to recover the shape and motion of pedestrians from sensor readings captured in the wild by a self-driving car driving around. We incorporate the reconstructed pedestrian assets bank in a realistic 3D simulation system. We show that the simulated LiDAR data can be used to significantly reduce the amount of real-world data required for visual perception tasks.
  arXiv  Detail & Related papers  (2020-11-16T17:16:32Z)
• LiDARsim: Realistic LiDAR Simulation by Leveraging the Real World [84.57894492587053]
  We develop a novel simulator that captures both the power of physics-based and learning-based simulation. We first utilize ray casting over the 3D scene and then use a deep neural network to produce deviations from the physics-based simulation. We showcase LiDARsim's usefulness for perception algorithms-testing on long-tail events and end-to-end closed-loop evaluation on safety-critical scenarios.
  arXiv  Detail & Related papers  (2020-06-16T17:44:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.