CMRNext: Camera to LiDAR Matching in the Wild for Localization and Extrinsic Calibration

• URL: http://arxiv.org/abs/2402.00129v3
• Date: Tue, 2 Apr 2024 12:42:42 GMT
• Title: CMRNext: Camera to LiDAR Matching in the Wild for Localization and Extrinsic Calibration
• Authors: Daniele Cattaneo, Abhinav Valada,
• Abstract summary: CMRNext is a novel approach for camera-LIDAR matching that is independent of sensor-specific parameters, generalizable, and can be used in the wild. We extensively evaluate CMRNext on six different robotic platforms, including three publicly available datasets and three in-house robots.
• Score: 9.693729708337125
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: LiDARs are widely used for mapping and localization in dynamic environments. However, their high cost limits their widespread adoption. On the other hand, monocular localization in LiDAR maps using inexpensive cameras is a cost-effective alternative for large-scale deployment. Nevertheless, most existing approaches struggle to generalize to new sensor setups and environments, requiring retraining or fine-tuning. In this paper, we present CMRNext, a novel approach for camera-LIDAR matching that is independent of sensor-specific parameters, generalizable, and can be used in the wild for monocular localization in LiDAR maps and camera-LiDAR extrinsic calibration. CMRNext exploits recent advances in deep neural networks for matching cross-modal data and standard geometric techniques for robust pose estimation. We reformulate the point-pixel matching problem as an optical flow estimation problem and solve the Perspective-n-Point problem based on the resulting correspondences to find the relative pose between the camera and the LiDAR point cloud. We extensively evaluate CMRNext on six different robotic platforms, including three publicly available datasets and three in-house robots. Our experimental evaluations demonstrate that CMRNext outperforms existing approaches on both tasks and effectively generalizes to previously unseen environments and sensor setups in a zero-shot manner. We make the code and pre-trained models publicly available at http://cmrnext.cs.uni-freiburg.de .

Related papers

• Gait Sequence Upsampling using Diffusion Models for Single LiDAR Sensors [1.0485739694839664]
  LidarGSU is designed to improve the generalization capability of existing identification models. In this work, we leverage DPMs on sparse sequential pedestrian point clouds as conditional masks in a video-to-video translation approach. We conduct extensive experiments on the SUSTeck1K dataset to evaluate the generative quality and recognition performance of the proposed method.
  arXiv  Detail & Related papers  (2024-10-11T10:11:21Z)
• ES-Gaussian: Gaussian Splatting Mapping via Error Space-Based Gaussian Completion [9.443354889048614]
  Vision-based mapping often struggles with high-quality 3D reconstruction due to sparse point clouds. We propose ES-Gaussian, an end-to-end system using a low-altitude camera and single-line LiDAR for high-quality 3D reconstruction.
  arXiv  Detail & Related papers  (2024-10-09T07:09:29Z)
• LiDAR-GS:Real-time LiDAR Re-Simulation using Gaussian Splatting [50.808933338389686]
  LiDAR simulation plays a crucial role in closed-loop simulation for autonomous driving. We present LiDAR-GS, the first LiDAR Gaussian Splatting method, for real-time high-fidelity re-simulation of LiDAR sensor scans in public urban road scenes. Our approach succeeds in simultaneously re-simulating depth, intensity, and ray-drop channels, achieving state-of-the-art results in both rendering frame rate and quality on publically available large scene datasets.
  arXiv  Detail & Related papers  (2024-10-07T15:07:56Z)
• LCPR: A Multi-Scale Attention-Based LiDAR-Camera Fusion Network for Place Recognition [11.206532393178385]
  We present a novel neural network named LCPR for robust multimodal place recognition. Our method can effectively utilize multi-view camera and LiDAR data to improve the place recognition performance.
  arXiv  Detail & Related papers  (2023-11-06T15:39:48Z)
• INF: Implicit Neural Fusion for LiDAR and Camera [7.123895040455239]
  Implicit neural fusion (INF) for LiDAR and camera is proposed in this paper. INF first trains a neural density field of the target scene using LiDAR frames. Then, a separate neural color field is trained using camera images and the trained neural density field.
  arXiv  Detail & Related papers  (2023-08-28T08:51:20Z)
• UnLoc: A Universal Localization Method for Autonomous Vehicles using LiDAR, Radar and/or Camera Input [51.150605800173366]
  UnLoc is a novel unified neural modeling approach for localization with multi-sensor input in all weather conditions. Our method is extensively evaluated on Oxford Radar RobotCar, ApolloSouthBay and Perth-WA datasets.
  arXiv  Detail & Related papers  (2023-07-03T04:10:55Z)
• LCE-Calib: Automatic LiDAR-Frame/Event Camera Extrinsic Calibration With A Globally Optimal Solution [10.117923901732743]
  The combination of LiDARs and cameras enables a mobile robot to perceive environments with multi-modal data. Traditional frame cameras are sensitive to changing illumination conditions, motivating us to introduce novel event cameras. This paper proposes an automatic checkerboard-based approach to calibrate extrinsics between a LiDAR and a frame/event camera.
  arXiv  Detail & Related papers  (2023-03-17T08:07:56Z)
• LiDARCap: Long-range Marker-less 3D Human Motion Capture with LiDAR Point Clouds [58.402752909624716]
  Existing motion capture datasets are largely short-range and cannot yet fit the need of long-range applications. We propose LiDARHuman26M, a new human motion capture dataset captured by LiDAR at a much longer range to overcome this limitation. Our dataset also includes the ground truth human motions acquired by the IMU system and the synchronous RGB images.
  arXiv  Detail & Related papers  (2022-03-28T12:52:45Z)
• Automatic Extrinsic Calibration Method for LiDAR and Camera Sensor Setups [68.8204255655161]
  We present a method to calibrate the parameters of any pair of sensors involving LiDARs, monocular or stereo cameras. The proposed approach can handle devices with very different resolutions and poses, as usually found in vehicle setups.
  arXiv  Detail & Related papers  (2021-01-12T12:02:26Z)
• Infrastructure-based Multi-Camera Calibration using Radial Projections [117.22654577367246]
  Pattern-based calibration techniques can be used to calibrate the intrinsics of the cameras individually. Infrastucture-based calibration techniques are able to estimate the extrinsics using 3D maps pre-built via SLAM or Structure-from-Motion. We propose to fully calibrate a multi-camera system from scratch using an infrastructure-based approach.
  arXiv  Detail & Related papers  (2020-07-30T09:21:04Z)
• 6D Camera Relocalization in Ambiguous Scenes via Continuous Multimodal Inference [67.70859730448473]
  We present a multimodal camera relocalization framework that captures ambiguities and uncertainties. We predict multiple camera pose hypotheses as well as the respective uncertainty for each prediction. We introduce a new dataset specifically designed to foster camera localization research in ambiguous environments.
  arXiv  Detail & Related papers  (2020-04-09T20:55:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.