VXP: Voxel-Cross-Pixel Large-scale Image-LiDAR Place Recognition

• URL: http://arxiv.org/abs/2403.14594v2
• Date: Fri, 14 Mar 2025 21:46:18 GMT
• Title: VXP: Voxel-Cross-Pixel Large-scale Image-LiDAR Place Recognition
• Authors: Yun-Jin Li, Mariia Gladkova, Yan Xia, Rui Wang, Daniel Cremers,
• Abstract summary: Cross-modal place recognition methods are flexible GPS-alternatives under varying environment conditions and sensor setups.<n>We propose Voxel-Cross-Pixel (VXP), a novel camera-to-LiDAR place recognition framework.<n>VXP enforces local similarities in a self-supervised manner and effectively brings global context from images and LiDAR scans into a shared feature space.
• Score: 40.603362112697255
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Cross-modal place recognition methods are flexible GPS-alternatives under varying environment conditions and sensor setups. However, this task is non-trivial since extracting consistent and robust global descriptors from different modalities is challenging. To tackle this issue, we propose Voxel-Cross-Pixel (VXP), a novel camera-to-LiDAR place recognition framework that enforces local similarities in a self-supervised manner and effectively brings global context from images and LiDAR scans into a shared feature space. Specifically, VXP is trained in three stages: first, we deploy a visual transformer to compactly represent input images. Secondly, we establish local correspondences between image-based and point cloud-based feature spaces using our novel geometric alignment module. We then aggregate local similarities into an expressive shared latent space. Extensive experiments on the three benchmarks (Oxford RobotCar, ViViD++ and KITTI) demonstrate that our method surpasses the state-of-the-art cross-modal retrieval by a large margin. Our evaluations show that the proposed method is accurate, efficient and light-weight. Our project page is available at: https://yunjinli.github.io/projects-vxp/

Related papers

• Circle-RoPE: Cone-like Decoupled Rotary Positional Embedding for Large Vision-Language Models [49.122200327049676]
  Rotary Position Embedding (RoPE) is a widely adopted technique for encoding relative positional information in large language models.<n>When extended to vision-language models (VLMs), RoPE and its variants enforce relative positional dependencies separately within text and image tokens.<n>We introduce Circle-RoPE, a novel encoding scheme designed to eliminate spurious cross-modal biases.
  arXiv  Detail & Related papers  (2025-05-22T09:05:01Z)
• A-SCoRe: Attention-based Scene Coordinate Regression for wide-ranging scenarios [1.2093553114715083]
  A-ScoRe is an Attention-based model which leverage attention on descriptor map level to produce meaningful and high-semantic 2D descriptors. Results show our methods achieve comparable performance with State-of-the-art methods on multiple benchmark while being light-weighted and much more flexible.
  arXiv  Detail & Related papers  (2025-03-18T07:39:50Z)
• Monocular Visual Place Recognition in LiDAR Maps via Cross-Modal State Space Model and Multi-View Matching [2.400446821380503]
  We introduce an efficient framework to learn descriptors for both RGB images and point clouds. It takes visual state space model (VMamba) as the backbone and employs a pixel-view-scene joint training strategy. A visible 3D points overlap strategy is then designed to quantify the similarity between point cloud views and RGB images for multi-view supervision.
  arXiv  Detail & Related papers  (2024-10-08T18:31:41Z)
• AddressCLIP: Empowering Vision-Language Models for City-wide Image Address Localization [57.34659640776723]
  We propose an end-to-end framework named AddressCLIP to solve the problem with more semantics. We have built three datasets from Pittsburgh and San Francisco on different scales specifically for the IAL problem.
  arXiv  Detail & Related papers  (2024-07-11T03:18:53Z)
• Context and Geometry Aware Voxel Transformer for Semantic Scene Completion [7.147020285382786]
  Vision-based Semantic Scene Completion (SSC) has gained much attention due to its widespread applications in various 3D perception tasks. Existing sparse-to-dense approaches typically employ shared context-independent queries across various input images. We introduce a neural network named CGFormer to achieve semantic scene completion.
  arXiv  Detail & Related papers  (2024-05-22T14:16:30Z)
• Differentiable Registration of Images and LiDAR Point Clouds with VoxelPoint-to-Pixel Matching [58.10418136917358]
  Cross-modality registration between 2D images from cameras and 3D point clouds from LiDARs is a crucial task in computer vision and robotic training. Previous methods estimate 2D-3D correspondences by matching point and pixel patterns learned by neural networks. We learn a structured cross-modality matching solver to represent 3D features via a different latent pixel space.
  arXiv  Detail & Related papers  (2023-12-07T05:46:10Z)
• PVT-SSD: Single-Stage 3D Object Detector with Point-Voxel Transformer [75.2251801053839]
  We present a novel Point-Voxel Transformer for single-stage 3D detection (PVT-SSD) We propose a Point-Voxel Transformer (PVT) module that obtains long-range contexts in a cheap manner from voxels. The experiments on several autonomous driving benchmarks verify the effectiveness and efficiency of the proposed method.
  arXiv  Detail & Related papers  (2023-05-11T07:37:15Z)
• Unleash the Potential of Image Branch for Cross-modal 3D Object Detection [67.94357336206136]
  We present a new cross-modal 3D object detector, namely UPIDet, which aims to unleash the potential of the image branch from two aspects. First, UPIDet introduces a new 2D auxiliary task called normalized local coordinate map estimation. Second, we discover that the representational capability of the point cloud backbone can be enhanced through the gradients backpropagated from the training objectives of the image branch.
  arXiv  Detail & Related papers  (2023-01-22T08:26:58Z)
• VPIT: Real-time Embedded Single Object 3D Tracking Using Voxel Pseudo Images [90.60881721134656]
  We propose a novel voxel-based 3D single object tracking (3D SOT) method called Voxel Pseudo Image Tracking (VPIT) Experiments on KITTI Tracking dataset show that VPIT is the fastest 3D SOT method and maintains competitive Success and Precision values.
  arXiv  Detail & Related papers  (2022-06-06T14:02:06Z)
• Unifying Voxel-based Representation with Transformer for 3D Object Detection [143.91910747605107]
  We present a unified framework for multi-modality 3D object detection, named UVTR. The proposed method aims to unify multi-modality representations in the voxel space for accurate and robust single- or cross-modality 3D detection. UVTR achieves leading performance in the nuScenes test set with 69.7%, 55.1%, and 71.1% NDS for LiDAR, camera, and multi-modality inputs, respectively.
  arXiv  Detail & Related papers  (2022-06-01T17:02:40Z)
• Exploring Intra- and Inter-Video Relation for Surgical Semantic Scene Segmentation [58.74791043631219]
  We propose a novel framework STswinCL that explores the complementary intra- and inter-video relations to boost segmentation performance. We extensively validate our approach on two public surgical video benchmarks, including EndoVis18 Challenge and CaDIS dataset. Experimental results demonstrate the promising performance of our method, which consistently exceeds previous state-of-the-art approaches.
  arXiv  Detail & Related papers  (2022-03-29T05:52:23Z)
• 3rd Place: A Global and Local Dual Retrieval Solution to Facebook AI Image Similarity Challenge [2.4340897078287815]
  This paper presents our 3rd place solution to the matching track of Image Similarity Challenge (ISC) 2021 organized by Facebook AI. We propose a multi-branch retrieval method of combining global descriptors and local descriptors to cover all attack cases. We show some ablation experiments of our method, which reveals the complementary advantages of global and local features.
  arXiv  Detail & Related papers  (2021-12-04T16:25:24Z)
• Progressive Coordinate Transforms for Monocular 3D Object Detection [52.00071336733109]
  We propose a novel and lightweight approach, dubbed em Progressive Coordinate Transforms (PCT) to facilitate learning coordinate representations. In this paper, we propose a novel and lightweight approach, dubbed em Progressive Coordinate Transforms (PCT) to facilitate learning coordinate representations.
  arXiv  Detail & Related papers  (2021-08-12T15:22:33Z)
• CORAL: Colored structural representation for bi-modal place recognition [12.357478978433814]
  We propose a bi-modal place recognition method, which can extract a compound global descriptor from the two modalities, vision and LiDAR. Specifically, we first build the elevation image generated from 3D points as a structural representation. Then, we derive the correspondences between 3D points and image pixels that are further used in merging the pixel-wise visual features into the elevation map grids.
  arXiv  Detail & Related papers  (2020-11-22T04:51:40Z)
• City-Scale Visual Place Recognition with Deep Local Features Based on Multi-Scale Ordered VLAD Pooling [5.274399407597545]
  We present a fully-automated system for place recognition at a city-scale based on content-based image retrieval. Firstly, we take a comprehensive analysis of visual place recognition and sketch out the unique challenges of the task. Next, we propose yet a simple pooling approach on top of convolutional neural network activations to embed the spatial information into the image representation vector.
  arXiv  Detail & Related papers  (2020-09-19T15:21:59Z)
• X-ModalNet: A Semi-Supervised Deep Cross-Modal Network for Classification of Remote Sensing Data [69.37597254841052]
  We propose a novel cross-modal deep-learning framework called X-ModalNet. X-ModalNet generalizes well, owing to propagating labels on an updatable graph constructed by high-level features on the top of the network. We evaluate X-ModalNet on two multi-modal remote sensing datasets (HSI-MSI and HSI-SAR) and achieve a significant improvement in comparison with several state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-06-24T15:29:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.