Scaling Diffusion Models to Real-World 3D LiDAR Scene Completion

• URL: http://arxiv.org/abs/2403.13470v1
• Date: Wed, 20 Mar 2024 10:19:05 GMT
• Title: Scaling Diffusion Models to Real-World 3D LiDAR Scene Completion
• Authors: Lucas Nunes, Rodrigo Marcuzzi, Benedikt Mersch, Jens Behley, Cyrill Stachniss,
• Abstract summary: 3D LiDAR sensors are commonly used to collect sparse 3D point clouds from the scene. We propose extending diffusion models as generative models for images to achieve scene completion from a single 3D LiDAR scan. Our method can complete the scene given a single LiDAR scan as input, producing a scene with more details compared to state-of-the-art scene completion methods.
• Score: 25.69896680908217
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Computer vision techniques play a central role in the perception stack of autonomous vehicles. Such methods are employed to perceive the vehicle surroundings given sensor data. 3D LiDAR sensors are commonly used to collect sparse 3D point clouds from the scene. However, compared to human perception, such systems struggle to deduce the unseen parts of the scene given those sparse point clouds. In this matter, the scene completion task aims at predicting the gaps in the LiDAR measurements to achieve a more complete scene representation. Given the promising results of recent diffusion models as generative models for images, we propose extending them to achieve scene completion from a single 3D LiDAR scan. Previous works used diffusion models over range images extracted from LiDAR data, directly applying image-based diffusion methods. Distinctly, we propose to directly operate on the points, reformulating the noising and denoising diffusion process such that it can efficiently work at scene scale. Together with our approach, we propose a regularization loss to stabilize the noise predicted during the denoising process. Our experimental evaluation shows that our method can complete the scene given a single LiDAR scan as input, producing a scene with more details compared to state-of-the-art scene completion methods. We believe that our proposed diffusion process formulation can support further research in diffusion models applied to scene-scale point cloud data.

Related papers

• Coherent 3D Scene Diffusion From a Single RGB Image [68.31336308924477]
  We present a novel diffusion-based approach for coherent 3D scene reconstruction from a single RGB image. Our method simultaneously denoises the 3D poses and geometries of all objects within the scene. By framing the task of single RGB image 3D scene reconstruction as a conditional diffusion process, our approach surpasses current state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-12-13T17:26:45Z)
• DiffSSC: Semantic LiDAR Scan Completion using Denoising Diffusion Probabilistic Models [18.342569823885864]
  3D LiDAR sensors are widely used to capture sparse point clouds of the vehicle's surroundings. Such systems struggle to perceive occluded areas and gaps in the scene due to the sparsity of these point clouds and their lack of semantics. We jointly predict unobserved geometry and semantics in the scene given raw LiDAR measurements, aiming for a more complete scene representation.
  arXiv  Detail & Related papers  (2024-09-26T17:39:05Z)
• SGD: Street View Synthesis with Gaussian Splatting and Diffusion Prior [53.52396082006044]
  Current methods struggle to maintain rendering quality at the viewpoint that deviates significantly from the training viewpoints. This issue stems from the sparse training views captured by a fixed camera on a moving vehicle. We propose a novel approach that enhances the capacity of 3DGS by leveraging prior from a Diffusion Model.
  arXiv  Detail & Related papers  (2024-03-29T09:20:29Z)
• OccNeRF: Advancing 3D Occupancy Prediction in LiDAR-Free Environments [77.0399450848749]
  We propose an OccNeRF method for training occupancy networks without 3D supervision. We parameterize the reconstructed occupancy fields and reorganize the sampling strategy to align with the cameras' infinite perceptive range. For semantic occupancy prediction, we design several strategies to polish the prompts and filter the outputs of a pretrained open-vocabulary 2D segmentation model.
  arXiv  Detail & Related papers  (2023-12-14T18:58:52Z)
• Diffusion-based 3D Object Detection with Random Boxes [58.43022365393569]
  Existing anchor-based 3D detection methods rely on empiricals setting of anchors, which makes the algorithms lack elegance. Our proposed Diff3Det migrates the diffusion model to proposal generation for 3D object detection by considering the detection boxes as generative targets. In the inference stage, the model progressively refines a set of random boxes to the prediction results.
  arXiv  Detail & Related papers  (2023-09-05T08:49:53Z)
• Diffusion with Forward Models: Solving Stochastic Inverse Problems Without Direct Supervision [76.32860119056964]
  We propose a novel class of denoising diffusion probabilistic models that learn to sample from distributions of signals that are never directly observed. We demonstrate the effectiveness of our method on three challenging computer vision tasks.
  arXiv  Detail & Related papers  (2023-06-20T17:53:00Z)
• Deceptive-NeRF/3DGS: Diffusion-Generated Pseudo-Observations for High-Quality Sparse-View Reconstruction [60.52716381465063]
  We introduce Deceptive-NeRF/3DGS to enhance sparse-view reconstruction with only a limited set of input images. Specifically, we propose a deceptive diffusion model turning noisy images rendered from few-view reconstructions into high-quality pseudo-observations. Our system progressively incorporates diffusion-generated pseudo-observations into the training image sets, ultimately densifying the sparse input observations by 5 to 10 times.
  arXiv  Detail & Related papers  (2023-05-24T14:00:32Z)
• Diffusion-based Generation, Optimization, and Planning in 3D Scenes [89.63179422011254]
  We introduce SceneDiffuser, a conditional generative model for 3D scene understanding. SceneDiffuser is intrinsically scene-aware, physics-based, and goal-oriented. We show significant improvements compared with previous models.
  arXiv  Detail & Related papers  (2023-01-15T03:43:45Z)
• DiffRF: Rendering-Guided 3D Radiance Field Diffusion [18.20324411024166]
  We introduce DiffRF, a novel approach for 3D radiance field synthesis based on denoising diffusion probabilistic models. In contrast to 2D-diffusion models, our model learns multi-view consistent priors, enabling free-view synthesis and accurate shape generation.
  arXiv  Detail & Related papers  (2022-12-02T14:37:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.