PAPI-Reg: Patch-to-Pixel Solution for Efficient Cross-Modal Registration between LiDAR Point Cloud and Camera Image

• URL: http://arxiv.org/abs/2503.15285v1
• Date: Wed, 19 Mar 2025 15:04:01 GMT
• Title: PAPI-Reg: Patch-to-Pixel Solution for Efficient Cross-Modal Registration between LiDAR Point Cloud and Camera Image
• Authors: Yuanchao Yue, Zhengxin Li, Wei Zhang, Hui Yuan,
• Abstract summary: Cross-modal data fusion involves the precise alignment of data from different sensors.<n>We propose a framework that projects point clouds into several 2D representations for matching with camera images.<n>To tackle the challenges of cross modal differences and the limited overlap between LiDAR point clouds and images in the image matching task, we introduce a multi-scale feature extraction network.
• Score: 10.906218491083576
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The primary requirement for cross-modal data fusion is the precise alignment of data from different sensors. However, the calibration between LiDAR point clouds and camera images is typically time-consuming and needs external calibration board or specific environmental features. Cross-modal registration effectively solves this problem by aligning the data directly without requiring external calibration. However, due to the domain gap between the point cloud and the image, existing methods rarely achieve satisfactory registration accuracy while maintaining real-time performance. To address this issue, we propose a framework that projects point clouds into several 2D representations for matching with camera images, which not only leverages the geometric characteristic of LiDAR point clouds more effectively but also bridge the domain gap between the point cloud and image. Moreover, to tackle the challenges of cross modal differences and the limited overlap between LiDAR point clouds and images in the image matching task, we introduce a multi-scale feature extraction network to effectively extract features from both camera images and the projection maps of LiDAR point cloud. Additionally, we propose a patch-to-pixel matching network to provide more effective supervision and achieve higher accuracy. We validate the performance of our model through experiments on the KITTI and nuScenes datasets. Our network achieves real-time performance and extremely high registration accuracy. On the KITTI dataset, our model achieves a registration accuracy rate of over 99\%.

Related papers

• EdgeRegNet: Edge Feature-based Multimodal Registration Network between Images and LiDAR Point Clouds [10.324549723042338]
  Cross-modal data registration has long been a critical task in computer vision.<n>We propose a method that uses edge information from the original point clouds and images for cross-modal registration.<n>We validate our method on the KITTI and nuScenes datasets, demonstrating its state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-03-19T15:03:41Z)
• LPRnet: A self-supervised registration network for LiDAR and photogrammetric point clouds [38.42527849407057]
  LiDAR and photogrammetry are active and passive remote sensing techniques for point cloud acquisition, respectively.<n>Due to the fundamental differences in sensing mechanisms, spatial distributions and coordinate systems, their point clouds exhibit significant discrepancies in density, precision, noise, and overlap.<n>This paper proposes a self-supervised registration network based on a masked autoencoder, focusing on heterogeneous LiDAR and photogrammetric point clouds.
  arXiv  Detail & Related papers  (2025-01-10T02:36:37Z)
• EEPNet: Efficient Edge Pixel-based Matching Network for Cross-Modal Dynamic Registration between LiDAR and Camera [6.817117737186402]
  Multisensor fusion is essential for autonomous vehicles to accurately perceive, analyze, and plan their trajectories within complex environments. Current methods for registering LiDAR point clouds with images face significant challenges due to inherent differences and computational overhead. We propose EEPNet, an advanced network that leverages modality maps obtained from point cloud projections to enhance registration accuracy.
  arXiv  Detail & Related papers  (2024-09-28T10:28:28Z)
• P2P-Bridge: Diffusion Bridges for 3D Point Cloud Denoising [81.92854168911704]
  We tackle the task of point cloud denoising through a novel framework that adapts Diffusion Schr"odinger bridges to points clouds. Experiments on object datasets show that P2P-Bridge achieves significant improvements over existing methods.
  arXiv  Detail & Related papers  (2024-08-29T08:00:07Z)
• Differentiable Registration of Images and LiDAR Point Clouds with VoxelPoint-to-Pixel Matching [58.10418136917358]
  Cross-modality registration between 2D images from cameras and 3D point clouds from LiDARs is a crucial task in computer vision and robotic training. Previous methods estimate 2D-3D correspondences by matching point and pixel patterns learned by neural networks. We learn a structured cross-modality matching solver to represent 3D features via a different latent pixel space.
  arXiv  Detail & Related papers  (2023-12-07T05:46:10Z)
• GraphAlign: Enhancing Accurate Feature Alignment by Graph matching for Multi-Modal 3D Object Detection [7.743525134435137]
  LiDAR and cameras are complementary sensors for 3D object detection in autonomous driving. We present GraphAlign, a more accurate feature alignment strategy for 3D object detection by graph matching.
  arXiv  Detail & Related papers  (2023-10-12T12:06:31Z)
• FreeReg: Image-to-Point Cloud Registration Leveraging Pretrained Diffusion Models and Monocular Depth Estimators [37.39693977657165]
  Matching cross-modality features between images and point clouds is a fundamental problem for image-to-point cloud registration. We propose to unify the modality between images and point clouds by pretrained large-scale models first. We show that the intermediate features, called diffusion features, extracted by depth-to-image diffusion models are semantically consistent between images and point clouds.
  arXiv  Detail & Related papers  (2023-10-05T09:57:23Z)
• CorrI2P: Deep Image-to-Point Cloud Registration via Dense Correspondence [51.91791056908387]
  We propose the first feature-based dense correspondence framework for addressing the image-to-point cloud registration problem, dubbed CorrI2P. Specifically, given a pair of a 2D image before a 3D point cloud, we first transform them into high-dimensional feature space feed the features into a symmetric overlapping region to determine the region where the image point cloud overlap.
  arXiv  Detail & Related papers  (2022-07-12T11:49:31Z)
• Unsupervised Foggy Scene Understanding via Self Spatial-Temporal Label Diffusion [51.11295961195151]
  We exploit the characteristics of the foggy image sequence of driving scenes to densify the confident pseudo labels. Based on the two discoveries of local spatial similarity and adjacent temporal correspondence of the sequential image data, we propose a novel Target-Domain driven pseudo label Diffusion scheme. Our scheme helps the adaptive model achieve 51.92% and 53.84% mean intersection-over-union (mIoU) on two publicly available natural foggy datasets.
  arXiv  Detail & Related papers  (2022-06-10T05:16:50Z)
• DeepI2P: Image-to-Point Cloud Registration via Deep Classification [71.3121124994105]
  DeepI2P is a novel approach for cross-modality registration between an image and a point cloud. Our method estimates the relative rigid transformation between the coordinate frames of the camera and Lidar. We circumvent the difficulty by converting the registration problem into a classification and inverse camera projection optimization problem.
  arXiv  Detail & Related papers  (2021-04-08T04:27:32Z)
• Pseudo-LiDAR Point Cloud Interpolation Based on 3D Motion Representation and Spatial Supervision [68.35777836993212]
  We propose a Pseudo-LiDAR point cloud network to generate temporally and spatially high-quality point cloud sequences. By exploiting the scene flow between point clouds, the proposed network is able to learn a more accurate representation of the 3D spatial motion relationship.
  arXiv  Detail & Related papers  (2020-06-20T03:11:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.