EdgeRegNet: Edge Feature-based Multimodal Registration Network between Images and LiDAR Point Clouds

• URL: http://arxiv.org/abs/2503.15284v1
• Date: Wed, 19 Mar 2025 15:03:41 GMT
• Title: EdgeRegNet: Edge Feature-based Multimodal Registration Network between Images and LiDAR Point Clouds
• Authors: Yuanchao Yue, Hui Yuan, Qinglong Miao, Xiaolong Mao, Raouf Hamzaoui, Peter Eisert,
• Abstract summary: Cross-modal data registration has long been a critical task in computer vision.<n>We propose a method that uses edge information from the original point clouds and images for cross-modal registration.<n>We validate our method on the KITTI and nuScenes datasets, demonstrating its state-of-the-art performance.
• Score: 10.324549723042338
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Cross-modal data registration has long been a critical task in computer vision, with extensive applications in autonomous driving and robotics. Accurate and robust registration methods are essential for aligning data from different modalities, forming the foundation for multimodal sensor data fusion and enhancing perception systems' accuracy and reliability. The registration task between 2D images captured by cameras and 3D point clouds captured by Light Detection and Ranging (LiDAR) sensors is usually treated as a visual pose estimation problem. High-dimensional feature similarities from different modalities are leveraged to identify pixel-point correspondences, followed by pose estimation techniques using least squares methods. However, existing approaches often resort to downsampling the original point cloud and image data due to computational constraints, inevitably leading to a loss in precision. Additionally, high-dimensional features extracted using different feature extractors from various modalities require specific techniques to mitigate cross-modal differences for effective matching. To address these challenges, we propose a method that uses edge information from the original point clouds and images for cross-modal registration. We retain crucial information from the original data by extracting edge points and pixels, enhancing registration accuracy while maintaining computational efficiency. The use of edge points and edge pixels allows us to introduce an attention-based feature exchange block to eliminate cross-modal disparities. Furthermore, we incorporate an optimal matching layer to improve correspondence identification. We validate the accuracy of our method on the KITTI and nuScenes datasets, demonstrating its state-of-the-art performance.

Related papers

• PAPI-Reg: Patch-to-Pixel Solution for Efficient Cross-Modal Registration between LiDAR Point Cloud and Camera Image [10.906218491083576]
  Cross-modal data fusion involves the precise alignment of data from different sensors.<n>We propose a framework that projects point clouds into several 2D representations for matching with camera images.<n>To tackle the challenges of cross modal differences and the limited overlap between LiDAR point clouds and images in the image matching task, we introduce a multi-scale feature extraction network.
  arXiv  Detail & Related papers  (2025-03-19T15:04:01Z)
• Differentiable Registration of Images and LiDAR Point Clouds with VoxelPoint-to-Pixel Matching [58.10418136917358]
  Cross-modality registration between 2D images from cameras and 3D point clouds from LiDARs is a crucial task in computer vision and robotic training. Previous methods estimate 2D-3D correspondences by matching point and pixel patterns learned by neural networks. We learn a structured cross-modality matching solver to represent 3D features via a different latent pixel space.
  arXiv  Detail & Related papers  (2023-12-07T05:46:10Z)
• FreeReg: Image-to-Point Cloud Registration Leveraging Pretrained Diffusion Models and Monocular Depth Estimators [37.39693977657165]
  Matching cross-modality features between images and point clouds is a fundamental problem for image-to-point cloud registration. We propose to unify the modality between images and point clouds by pretrained large-scale models first. We show that the intermediate features, called diffusion features, extracted by depth-to-image diffusion models are semantically consistent between images and point clouds.
  arXiv  Detail & Related papers  (2023-10-05T09:57:23Z)
• Two Approaches to Supervised Image Segmentation [55.616364225463066]
  The present work develops comparison experiments between deep learning and multiset neurons approaches. The deep learning approach confirmed its potential for performing image segmentation. The alternative multiset methodology allowed for enhanced accuracy while requiring little computational resources.
  arXiv  Detail & Related papers  (2023-07-19T16:42:52Z)
• Quantity-Aware Coarse-to-Fine Correspondence for Image-to-Point Cloud Registration [4.954184310509112]
  Image-to-point cloud registration aims to determine the relative camera pose between an RGB image and a reference point cloud. Matching individual points with pixels can be inherently ambiguous due to modality gaps. We propose a framework to capture quantity-aware correspondences between local point sets and pixel patches.
  arXiv  Detail & Related papers  (2023-07-14T03:55:54Z)
• Image-to-Lidar Self-Supervised Distillation for Autonomous Driving Data [80.14669385741202]
  We propose a self-supervised pre-training method for 3D perception models tailored to autonomous driving data. We leverage the availability of synchronized and calibrated image and Lidar sensors in autonomous driving setups. Our method does not require any point cloud nor image annotations.
  arXiv  Detail & Related papers  (2022-03-30T12:40:30Z)
• A Robust Multimodal Remote Sensing Image Registration Method and System Using Steerable Filters with First- and Second-order Gradients [7.813406811407584]
  Co-registration of multimodal remote sensing images is still an ongoing challenge because of nonlinear radiometric differences (NRD) and significant geometric distortions. In this paper, a robust matching method based on the Steerable filters is proposed consisting of two critical steps. The performance of the proposed matching method has been evaluated with many different kinds of multimodal RS images.
  arXiv  Detail & Related papers  (2022-02-27T12:22:42Z)
• EPMF: Efficient Perception-aware Multi-sensor Fusion for 3D Semantic Segmentation [62.210091681352914]
  We study multi-sensor fusion for 3D semantic segmentation for many applications, such as autonomous driving and robotics. In this work, we investigate a collaborative fusion scheme called perception-aware multi-sensor fusion (PMF) We propose a two-stream network to extract features from the two modalities separately. The extracted features are fused by effective residual-based fusion modules.
  arXiv  Detail & Related papers  (2021-06-21T10:47:26Z)
• LocalTrans: A Multiscale Local Transformer Network for Cross-Resolution Homography Estimation [52.63874513999119]
  Cross-resolution image alignment is a key problem in multiscale giga photography. Existing deep homography methods neglecting the explicit formulation of correspondences between them, which leads to degraded accuracy in cross-resolution challenges. We propose a local transformer network embedded within a multiscale structure to explicitly learn correspondences between the multimodal inputs.
  arXiv  Detail & Related papers  (2021-06-08T02:51:45Z)
• DeepI2P: Image-to-Point Cloud Registration via Deep Classification [71.3121124994105]
  DeepI2P is a novel approach for cross-modality registration between an image and a point cloud. Our method estimates the relative rigid transformation between the coordinate frames of the camera and Lidar. We circumvent the difficulty by converting the registration problem into a classification and inverse camera projection optimization problem.
  arXiv  Detail & Related papers  (2021-04-08T04:27:32Z)
• Identity-Aware Attribute Recognition via Real-Time Distributed Inference in Mobile Edge Clouds [53.07042574352251]
  We design novel models for pedestrian attribute recognition with re-ID in an MEC-enabled camera monitoring system. We propose a novel inference framework with a set of distributed modules, by jointly considering the attribute recognition and person re-ID. We then devise a learning-based algorithm for the distributions of the modules of the proposed distributed inference framework.
  arXiv  Detail & Related papers  (2020-08-12T12:03:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.