Real-time Neural Rendering of LiDAR Point Clouds

• URL: http://arxiv.org/abs/2502.11618v1
• Date: Mon, 17 Feb 2025 10:01:13 GMT
• Title: Real-time Neural Rendering of LiDAR Point Clouds
• Authors: Joni Vanherck, Brent Zoomers, Tom Mertens, Lode Jorissen, Nick Michiels,
• Abstract summary: A naive projection of the point cloud to the output view using 1x1 pixels is fast and retains the available detail, but also results in unintelligible renderings as background points leak in between the foreground pixels.<n>A deep convolutional model in the form of a U-Net is used to transform these projections into a realistic result.<n>We also describe a method to generate synthetic training data to deal with imperfectly-aligned ground truth images.
• Score: 0.2621434923709917
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Static LiDAR scanners produce accurate, dense, colored point clouds, but often contain obtrusive artifacts which makes them ill-suited for direct display. We propose an efficient method to render photorealistic images of such scans without any expensive preprocessing or training of a scene-specific model. A naive projection of the point cloud to the output view using 1x1 pixels is fast and retains the available detail, but also results in unintelligible renderings as background points leak in between the foreground pixels. The key insight is that these projections can be transformed into a realistic result using a deep convolutional model in the form of a U-Net, and a depth-based heuristic that prefilters the data. The U-Net also handles LiDAR-specific problems such as missing parts due to occlusion, color inconsistencies and varying point densities. We also describe a method to generate synthetic training data to deal with imperfectly-aligned ground truth images. Our method achieves real-time rendering rates using an off-the-shelf GPU and outperforms the state-of-the-art in both speed and quality.

Related papers

• Real-time Free-view Human Rendering from Sparse-view RGB Videos using Double Unprojected Textures [87.80984588545589]
  Real-time free-view human rendering from sparse-view RGB inputs is a challenging task due to the sensor scarcity and the tight time budget.<n>Recent methods leverage 2D CNNs operating in texture space to learn rendering primitives.<n>We present Double Unprojected Textures, which at the core disentangles coarse geometric deformation estimation from appearance synthesis.
  arXiv  Detail & Related papers  (2024-12-17T18:57:38Z)
• EVER: Exact Volumetric Ellipsoid Rendering for Real-time View Synthesis [72.53316783628803]
  We present Exact Volumetric Ellipsoid Rendering (EVER), a method for real-time differentiable emission-only volume rendering. Unlike recentization based approach by 3D Gaussian Splatting (3DGS), our primitive based representation allows for exact volume rendering. We show that our method is more accurate with blending issues than 3DGS and follow-up work on view rendering.
  arXiv  Detail & Related papers  (2024-10-02T17:59:09Z)
• Inverse Neural Rendering for Explainable Multi-Object Tracking [35.072142773300655]
  We recast 3D multi-object tracking from RGB cameras as an emphInverse Rendering (IR) problem. We optimize an image loss over generative latent spaces that inherently disentangle shape and appearance properties. We validate the generalization and scaling capabilities of our method by learning the generative prior exclusively from synthetic data.
  arXiv  Detail & Related papers  (2024-04-18T17:37:53Z)
• Intrinsic Image Decomposition Using Point Cloud Representation [13.771632868567277]
  We introduce Point Intrinsic Net (PoInt-Net), which leverages 3D point cloud data to concurrently estimate albedo and shading maps. PoInt-Net is efficient, achieving consistent performance across point clouds of any size with training only required on small-scale point clouds.
  arXiv  Detail & Related papers  (2023-07-20T14:51:28Z)
• Multiscale Representation for Real-Time Anti-Aliasing Neural Rendering [84.37776381343662]
  Mip-NeRF proposes a multiscale representation as a conical frustum to encode scale information. We propose mip voxel grids (Mip-VoG), an explicit multiscale representation for real-time anti-aliasing rendering. Our approach is the first to offer multiscale training and real-time anti-aliasing rendering simultaneously.
  arXiv  Detail & Related papers  (2023-04-20T04:05:22Z)
• Point2Pix: Photo-Realistic Point Cloud Rendering via Neural Radiance Fields [63.21420081888606]
  Recent Radiance Fields and extensions are proposed to synthesize realistic images from 2D input. We present Point2Pix as a novel point to link the 3D sparse point clouds with 2D dense image pixels.
  arXiv  Detail & Related papers  (2023-03-29T06:26:55Z)
• Efficient Implicit Neural Reconstruction Using LiDAR [6.516471975863534]
  We propose a new method that uses sparse LiDAR point clouds and rough odometry to reconstruct fine-grained implicit occupancy field efficiently within a few minutes. As far as we know, our method is the first to reconstruct implicit scene representation from LiDAR-only input.
  arXiv  Detail & Related papers  (2023-02-28T07:31:48Z)
• Generating Photo-realistic Images from LiDAR Point Clouds with Generative Adversarial Networks [0.0]
  We created a dataset of point cloud image pairs and trained the GAN to predict images from LiDAR point clouds containing reflectance and distance information. Our models learned how to predict realistically looking images from just point cloud data, even images with black cars.
  arXiv  Detail & Related papers  (2021-12-20T05:25:15Z)
• Object-based Illumination Estimation with Rendering-aware Neural Networks [56.01734918693844]
  We present a scheme for fast environment light estimation from the RGBD appearance of individual objects and their local image areas. With the estimated lighting, virtual objects can be rendered in AR scenarios with shading that is consistent to the real scene.
  arXiv  Detail & Related papers  (2020-08-06T08:23:19Z)
• Intrinsic Autoencoders for Joint Neural Rendering and Intrinsic Image Decomposition [67.9464567157846]
  We propose an autoencoder for joint generation of realistic images from synthetic 3D models while simultaneously decomposing real images into their intrinsic shape and appearance properties. Our experiments confirm that a joint treatment of rendering and decomposition is indeed beneficial and that our approach outperforms state-of-the-art image-to-image translation baselines both qualitatively and quantitatively.
  arXiv  Detail & Related papers  (2020-06-29T12:53:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.