Blurred LiDAR for Sharper 3D: Robust Handheld 3D Scanning with Diffuse LiDAR and RGB

• URL: http://arxiv.org/abs/2411.19474v1
• Date: Fri, 29 Nov 2024 05:01:23 GMT
• Title: Blurred LiDAR for Sharper 3D: Robust Handheld 3D Scanning with Diffuse LiDAR and RGB
• Authors: Nikhil Behari, Aaron Young, Siddharth Somasundaram, Tzofi Klinghoffer, Akshat Dave, Ramesh Raskar,
• Abstract summary: 3D surface reconstruction is essential across applications of virtual reality, robotics, and mobile scanning. RGB-based reconstruction often fails in low-texture, low-light, and low-albedo scenes. We propose using an alternative class of "blurred" LiDAR that emits a diffuse flash.
• Score: 12.38882701862349
• License:
• Abstract: 3D surface reconstruction is essential across applications of virtual reality, robotics, and mobile scanning. However, RGB-based reconstruction often fails in low-texture, low-light, and low-albedo scenes. Handheld LiDARs, now common on mobile devices, aim to address these challenges by capturing depth information from time-of-flight measurements of a coarse grid of projected dots. Yet, these sparse LiDARs struggle with scene coverage on limited input views, leaving large gaps in depth information. In this work, we propose using an alternative class of "blurred" LiDAR that emits a diffuse flash, greatly improving scene coverage but introducing spatial ambiguity from mixed time-of-flight measurements across a wide field of view. To handle these ambiguities, we propose leveraging the complementary strengths of diffuse LiDAR with RGB. We introduce a Gaussian surfel-based rendering framework with a scene-adaptive loss function that dynamically balances RGB and diffuse LiDAR signals. We demonstrate that, surprisingly, diffuse LiDAR can outperform traditional sparse LiDAR, enabling robust 3D scanning with accurate color and geometry estimation in challenging environments.

Related papers

• ES-Gaussian: Gaussian Splatting Mapping via Error Space-Based Gaussian Completion [9.443354889048614]
  Vision-based mapping often struggles with high-quality 3D reconstruction due to sparse point clouds. We propose ES-Gaussian, an end-to-end system using a low-altitude camera and single-line LiDAR for high-quality 3D reconstruction.
  arXiv  Detail & Related papers  (2024-10-09T07:09:29Z)
• LiDAR-GS:Real-time LiDAR Re-Simulation using Gaussian Splatting [50.808933338389686]
  LiDAR simulation plays a crucial role in closed-loop simulation for autonomous driving. We present LiDAR-GS, the first LiDAR Gaussian Splatting method, for real-time high-fidelity re-simulation of LiDAR sensor scans in public urban road scenes. Our approach succeeds in simultaneously re-simulating depth, intensity, and ray-drop channels, achieving state-of-the-art results in both rendering frame rate and quality on publically available large scene datasets.
  arXiv  Detail & Related papers  (2024-10-07T15:07:56Z)
• TCLC-GS: Tightly Coupled LiDAR-Camera Gaussian Splatting for Autonomous Driving [14.80202289008908]
  Most 3D Gaussian Splatting (3D-GS) based methods for urban scenes initialize 3D Gaussians directly with 3D LiDAR points. We design a novel LiDAR-Camera Gaussian Splatting (TCLC-GS) to fully leverage the combined strengths of both LiDAR and camera sensors. Our method demonstrates fast training and achieves real-time RGB and depth rendering at 90 FPS in resolution of 1920x1280 (Waymo) and 120 FPS in resolution of 1600x900 (nuScenes) in urban scenarios.
  arXiv  Detail & Related papers  (2024-04-03T02:26:15Z)
• DeepMIF: Deep Monotonic Implicit Fields for Large-Scale LiDAR 3D Mapping [46.80755234561584]
  Recent learning-based methods integrate neural implicit representations and optimizable feature grids to approximate surfaces of 3D scenes. In this work we depart from fitting LiDAR data exactly, instead letting the network optimize a non-metric monotonic implicit field defined in 3D space. Our algorithm achieves high-quality dense 3D mapping performance as captured by multiple quantitative and perceptual measures and visual results obtained for Mai City, Newer College, and KITTI benchmarks.
  arXiv  Detail & Related papers  (2024-03-26T09:58:06Z)
• CVT-xRF: Contrastive In-Voxel Transformer for 3D Consistent Radiance Fields from Sparse Inputs [65.80187860906115]
  We propose a novel approach to improve NeRF's performance with sparse inputs. We first adopt a voxel-based ray sampling strategy to ensure that the sampled rays intersect with a certain voxel in 3D space. We then randomly sample additional points within the voxel and apply a Transformer to infer the properties of other points on each ray, which are then incorporated into the volume rendering.
  arXiv  Detail & Related papers  (2024-03-25T15:56:17Z)
• UltraLiDAR: Learning Compact Representations for LiDAR Completion and Generation [51.443788294845845]
  We present UltraLiDAR, a data-driven framework for scene-level LiDAR completion, LiDAR generation, and LiDAR manipulation. We show that by aligning the representation of a sparse point cloud to that of a dense point cloud, we can densify the sparse point clouds. By learning a prior over the discrete codebook, we can generate diverse, realistic LiDAR point clouds for self-driving.
  arXiv  Detail & Related papers  (2023-11-02T17:57:03Z)
• LiDAR-NeRF: Novel LiDAR View Synthesis via Neural Radiance Fields [112.62936571539232]
  We introduce a new task, novel view synthesis for LiDAR sensors. Traditional model-based LiDAR simulators with style-transfer neural networks can be applied to render novel views. We use a neural radiance field (NeRF) to facilitate the joint learning of geometry and the attributes of 3D points.
  arXiv  Detail & Related papers  (2023-04-20T15:44:37Z)
• LiDAR Distillation: Bridging the Beam-Induced Domain Gap for 3D Object Detection [96.63947479020631]
  In many real-world applications, the LiDAR points used by mass-produced robots and vehicles usually have fewer beams than that in large-scale public datasets. We propose the LiDAR Distillation to bridge the domain gap induced by different LiDAR beams for 3D object detection.
  arXiv  Detail & Related papers  (2022-03-28T17:59:02Z)
• LiDARCap: Long-range Marker-less 3D Human Motion Capture with LiDAR Point Clouds [58.402752909624716]
  Existing motion capture datasets are largely short-range and cannot yet fit the need of long-range applications. We propose LiDARHuman26M, a new human motion capture dataset captured by LiDAR at a much longer range to overcome this limitation. Our dataset also includes the ground truth human motions acquired by the IMU system and the synchronous RGB images.
  arXiv  Detail & Related papers  (2022-03-28T12:52:45Z)
• Consistent Depth Prediction under Various Illuminations using Dilated Cross Attention [1.332560004325655]
  We propose to use internet 3D indoor scenes and manually tune their illuminations to render photo-realistic RGB photos and their corresponding depth and BRDF maps. We perform cross attention on these dilated features to retain the consistency of depth prediction under different illuminations. Our method is evaluated by comparing it with current state-of-the-art methods on Vari dataset and a significant improvement is observed in experiments.
  arXiv  Detail & Related papers  (2021-12-15T10:02:46Z)
• DeepLiDARFlow: A Deep Learning Architecture For Scene Flow Estimation Using Monocular Camera and Sparse LiDAR [10.303618438296981]
  Scene flow is the dense 3D reconstruction of motion and geometry of a scene. Most state-of-the-art methods use a pair of stereo images as input for full scene reconstruction. DeepLiDARFlow is a novel deep learning architecture which fuses high level RGB and LiDAR features at multiple scales.
  arXiv  Detail & Related papers  (2020-08-18T19:51:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.