INF: Implicit Neural Fusion for LiDAR and Camera

• URL: http://arxiv.org/abs/2308.14414v1
• Date: Mon, 28 Aug 2023 08:51:20 GMT
• Title: INF: Implicit Neural Fusion for LiDAR and Camera
• Authors: Shuyi Zhou, Shuxiang Xie, Ryoichi Ishikawa, Ken Sakurada, Masaki Onishi, Takeshi Oishi
• Abstract summary: Implicit neural fusion (INF) for LiDAR and camera is proposed in this paper. INF first trains a neural density field of the target scene using LiDAR frames. Then, a separate neural color field is trained using camera images and the trained neural density field.
• Score: 7.123895040455239
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Sensor fusion has become a popular topic in robotics. However, conventional fusion methods encounter many difficulties, such as data representation differences, sensor variations, and extrinsic calibration. For example, the calibration methods used for LiDAR-camera fusion often require manual operation and auxiliary calibration targets. Implicit neural representations (INRs) have been developed for 3D scenes, and the volume density distribution involved in an INR unifies the scene information obtained by different types of sensors. Therefore, we propose implicit neural fusion (INF) for LiDAR and camera. INF first trains a neural density field of the target scene using LiDAR frames. Then, a separate neural color field is trained using camera images and the trained neural density field. Along with the training process, INF both estimates LiDAR poses and optimizes extrinsic parameters. Our experiments demonstrate the high accuracy and stable performance of the proposed method.

Related papers

• SGD: Street View Synthesis with Gaussian Splatting and Diffusion Prior [53.52396082006044]
  Current methods struggle to maintain rendering quality at the viewpoint that deviates significantly from the training viewpoints. This issue stems from the sparse training views captured by a fixed camera on a moving vehicle. We propose a novel approach that enhances the capacity of 3DGS by leveraging prior from a Diffusion Model.
  arXiv  Detail & Related papers  (2024-03-29T09:20:29Z)
• URS-NeRF: Unordered Rolling Shutter Bundle Adjustment for Neural Radiance Fields [46.186869281594326]
  We propose a novel rolling shutter bundle adjustment method for neural radiance fields (NeRF) We use the unordered rolling shutter (RS) images to obtain the implicit 3D representation.
  arXiv  Detail & Related papers  (2024-03-15T09:08:27Z)
• CMRNext: Camera to LiDAR Matching in the Wild for Localization and Extrinsic Calibration [9.693729708337125]
  CMRNext is a novel approach for camera-LIDAR matching that is independent of sensor-specific parameters, generalizable, and can be used in the wild. We extensively evaluate CMRNext on six different robotic platforms, including three publicly available datasets and three in-house robots.
  arXiv  Detail & Related papers  (2024-01-31T19:14:12Z)
• Leveraging Neural Radiance Fields for Uncertainty-Aware Visual Localization [56.95046107046027]
  We propose to leverage Neural Radiance Fields (NeRF) to generate training samples for scene coordinate regression. Despite NeRF's efficiency in rendering, many of the rendered data are polluted by artifacts or only contain minimal information gain.
  arXiv  Detail & Related papers  (2023-10-10T20:11:13Z)
• UnLoc: A Universal Localization Method for Autonomous Vehicles using LiDAR, Radar and/or Camera Input [51.150605800173366]
  UnLoc is a novel unified neural modeling approach for localization with multi-sensor input in all weather conditions. Our method is extensively evaluated on Oxford Radar RobotCar, ApolloSouthBay and Perth-WA datasets.
  arXiv  Detail & Related papers  (2023-07-03T04:10:55Z)
• UncLe-SLAM: Uncertainty Learning for Dense Neural SLAM [60.575435353047304]
  We present an uncertainty learning framework for dense neural simultaneous localization and mapping (SLAM) We propose an online framework for sensor uncertainty estimation that can be trained in a self-supervised manner from only 2D input data.
  arXiv  Detail & Related papers  (2023-06-19T16:26:25Z)
• Multi-task Learning for Camera Calibration [3.274290296343038]
  We present a unique method for predicting intrinsic (principal point offset and focal length) and extrinsic (baseline, pitch, and translation) properties from a pair of images. By reconstructing the 3D points using a camera model neural network and then using the loss in reconstruction to obtain the camera specifications, this innovative camera projection loss (CPL) method allows us that the desired parameters should be estimated.
  arXiv  Detail & Related papers  (2022-11-22T17:39:31Z)
• CLONeR: Camera-Lidar Fusion for Occupancy Grid-aided Neural Representations [77.90883737693325]
  This paper proposes CLONeR, which significantly improves upon NeRF by allowing it to model large outdoor driving scenes observed from sparse input sensor views. This is achieved by decoupling occupancy and color learning within the NeRF framework into separate Multi-Layer Perceptrons (MLPs) trained using LiDAR and camera data, respectively. In addition, this paper proposes a novel method to build differentiable 3D Occupancy Grid Maps (OGM) alongside the NeRF model, and leverage this occupancy grid for improved sampling of points along a ray for rendering in metric space.
  arXiv  Detail & Related papers  (2022-09-02T17:44:50Z)
• RVMDE: Radar Validated Monocular Depth Estimation for Robotics [5.360594929347198]
  An innate rigid calibration of binocular vision sensors is crucial for accurate depth estimation. Alternatively, a monocular camera alleviates the limitation at the expense of accuracy in estimating depth, and the challenge exacerbates in harsh environmental conditions. This work explores the utility of coarse signals from radar when fused with fine-grained data from a monocular camera for depth estimation in harsh environmental conditions.
  arXiv  Detail & Related papers  (2021-09-11T12:02:29Z)
• CalibDNN: Multimodal Sensor Calibration for Perception Using Deep Neural Networks [27.877734292570967]
  We propose a novel deep learning-driven technique (CalibDNN) for accurate calibration among multimodal sensor, specifically LiDAR-Camera pairs. The entire processing is fully automatic with a single model and single iteration. Results comparison among different methods and extensive experiments on different datasets demonstrates the state-of-the-art performance.
  arXiv  Detail & Related papers  (2021-03-27T02:43:37Z)
• Learning Camera Miscalibration Detection [83.38916296044394]
  This paper focuses on a data-driven approach to learn the detection of miscalibration in vision sensors, specifically RGB cameras. Our contributions include a proposed miscalibration metric for RGB cameras and a novel semi-synthetic dataset generation pipeline based on this metric. By training a deep convolutional neural network, we demonstrate the effectiveness of our pipeline to identify whether a recalibration of the camera's intrinsic parameters is required or not.
  arXiv  Detail & Related papers  (2020-05-24T10:32:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.