2D3D-MATR: 2D-3D Matching Transformer for Detection-free Registration between Images and Point Clouds

• URL: http://arxiv.org/abs/2308.05667v2
• Date: Mon, 14 Aug 2023 12:49:28 GMT
• Title: 2D3D-MATR: 2D-3D Matching Transformer for Detection-free Registration between Images and Point Clouds
• Authors: Minhao Li, Zheng Qin, Zhirui Gao, Renjiao Yi, Chenyang Zhu, Yulan Guo, Kai Xu
• Abstract summary: We propose 2D3D-MATR, a detection-free method for accurate and robust registration between images and point clouds. Our method adopts a coarse-to-fine pipeline where it first computes coarse correspondences between downsampled patches of the input image and the point cloud. To resolve the scale ambiguity in patch matching, we construct a multi-scale pyramid for each image patch and learn to find for each point patch the best matching image patch at a proper resolution level.
• Score: 38.425876064671435
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The commonly adopted detect-then-match approach to registration finds difficulties in the cross-modality cases due to the incompatible keypoint detection and inconsistent feature description. We propose, 2D3D-MATR, a detection-free method for accurate and robust registration between images and point clouds. Our method adopts a coarse-to-fine pipeline where it first computes coarse correspondences between downsampled patches of the input image and the point cloud and then extends them to form dense correspondences between pixels and points within the patch region. The coarse-level patch matching is based on transformer which jointly learns global contextual constraints with self-attention and cross-modality correlations with cross-attention. To resolve the scale ambiguity in patch matching, we construct a multi-scale pyramid for each image patch and learn to find for each point patch the best matching image patch at a proper resolution level. Extensive experiments on two public benchmarks demonstrate that 2D3D-MATR outperforms the previous state-of-the-art P2-Net by around $20$ percentage points on inlier ratio and over $10$ points on registration recall. Our code and models are available at https://github.com/minhaolee/2D3DMATR.

Related papers

• Robust 3D Point Clouds Classification based on Declarative Defenders [18.51700931775295]
  3D point clouds are unstructured and sparse, while 2D images are structured and dense. In this paper, we explore three distinct algorithms for mapping 3D point clouds into 2D images. The proposed approaches demonstrate superior accuracy and robustness against adversarial attacks.
  arXiv  Detail & Related papers  (2024-10-13T01:32:38Z)
• RetinaRegNet: A Zero-Shot Approach for Retinal Image Registration [10.430563602981705]
  RetinaRegNet is a zero-shot registration model designed to register retinal images with minimal overlap, large deformations, and varying image quality. We implement a two-stage registration framework to handle large deformations. Our model consistently outperformed state-of-the-art methods across all datasets.
  arXiv  Detail & Related papers  (2024-04-24T17:50:37Z)
• EP2P-Loc: End-to-End 3D Point to 2D Pixel Localization for Large-Scale Visual Localization [44.05930316729542]
  We propose EP2P-Loc, a novel large-scale visual localization method for 3D point clouds. To increase the number of inliers, we propose a simple algorithm to remove invisible 3D points in the image. For the first time in this task, we employ a differentiable for end-to-end training.
  arXiv  Detail & Related papers  (2023-09-14T07:06:36Z)
• CorrI2P: Deep Image-to-Point Cloud Registration via Dense Correspondence [51.91791056908387]
  We propose the first feature-based dense correspondence framework for addressing the image-to-point cloud registration problem, dubbed CorrI2P. Specifically, given a pair of a 2D image before a 3D point cloud, we first transform them into high-dimensional feature space feed the features into a symmetric overlapping region to determine the region where the image point cloud overlap.
  arXiv  Detail & Related papers  (2022-07-12T11:49:31Z)
• Geometric Transformer for Fast and Robust Point Cloud Registration [53.10568889775553]
  We study the problem of extracting accurate correspondences for point cloud registration. Recent keypoint-free methods bypass the detection of repeatable keypoints which is difficult in low-overlap scenarios. We propose Geometric Transformer to learn geometric feature for robust superpoint matching.
  arXiv  Detail & Related papers  (2022-02-14T13:26:09Z)
• DeepI2P: Image-to-Point Cloud Registration via Deep Classification [71.3121124994105]
  DeepI2P is a novel approach for cross-modality registration between an image and a point cloud. Our method estimates the relative rigid transformation between the coordinate frames of the camera and Lidar. We circumvent the difficulty by converting the registration problem into a classification and inverse camera projection optimization problem.
  arXiv  Detail & Related papers  (2021-04-08T04:27:32Z)
• P2-Net: Joint Description and Detection of Local Features for Pixel and Point Matching [78.18641868402901]
  This work takes the initiative to establish fine-grained correspondences between 2D images and 3D point clouds. An ultra-wide reception mechanism in combination with a novel loss function are designed to mitigate the intrinsic information variations between pixel and point local regions.
  arXiv  Detail & Related papers  (2021-03-01T14:59:40Z)
• Learning 2D-3D Correspondences To Solve The Blind Perspective-n-Point Problem [98.92148855291363]
  This paper proposes a deep CNN model which simultaneously solves for both 6-DoF absolute camera pose 2D--3D correspondences. Tests on both real and simulated data have shown that our method substantially outperforms existing approaches.
  arXiv  Detail & Related papers  (2020-03-15T04:17:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.