BEV$^2$PR: BEV-Enhanced Visual Place Recognition with Structural Cues

• URL: http://arxiv.org/abs/2403.06600v2
• Date: Tue, 23 Jul 2024 12:20:27 GMT
• Title: BEV$^2$PR: BEV-Enhanced Visual Place Recognition with Structural Cues
• Authors: Fudong Ge, Yiwei Zhang, Shuhan Shen, Yue Wang, Weiming Hu, Jin Gao,
• Abstract summary: We propose a new image-based visual place recognition (VPR) framework by exploiting the structural cues in bird's-eye view (BEV) from a single camera. The BEV$2$PR framework generates a composite descriptor with both visual cues and spatial awareness based on a single camera.
• Score: 44.96177875644304
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper, we propose a new image-based visual place recognition (VPR) framework by exploiting the structural cues in bird's-eye view (BEV) from a single monocular camera. The motivation arises from two key observations about place recognition methods based on both appearance and structure: 1) For the methods relying on LiDAR sensors, the integration of LiDAR in robotic systems has led to increased expenses, while the alignment of data between different sensors is also a major challenge. 2) Other image-/camera-based methods, involving integrating RGB images and their derived variants (eg, pseudo depth images, pseudo 3D point clouds), exhibit several limitations, such as the failure to effectively exploit the explicit spatial relationships between different objects. To tackle the above issues, we design a new BEV-enhanced VPR framework, namely BEV$^2$PR, generating a composite descriptor with both visual cues and spatial awareness based on a single camera. The key points lie in: 1) We use BEV features as an explicit source of structural knowledge in constructing global features. 2) The lower layers of the pre-trained backbone from BEV generation are shared for visual and structural streams in VPR, facilitating the learning of fine-grained local features in the visual stream. 3) The complementary visual and structural features can jointly enhance VPR performance. Our BEV$^2$PR framework enables consistent performance improvements over several popular aggregation modules for RGB global features. The experiments on our collected VPR-NuScenes dataset demonstrate an absolute gain of 2.47% on Recall@1 for the strong Conv-AP baseline to achieve the best performance in our setting, and notably, a 18.06% gain on the hard set. The code and dataset will be available at https://github.com/FudongGe/BEV2PR.

Related papers

• OE-BevSeg: An Object Informed and Environment Aware Multimodal Framework for Bird's-eye-view Vehicle Semantic Segmentation [57.2213693781672]
  Bird's-eye-view (BEV) semantic segmentation is becoming crucial in autonomous driving systems. We propose OE-BevSeg, an end-to-end multimodal framework that enhances BEV segmentation performance. Our approach achieves state-of-the-art results by a large margin on the nuScenes dataset for vehicle segmentation.
  arXiv  Detail & Related papers  (2024-07-18T03:48:22Z)
• IFTR: An Instance-Level Fusion Transformer for Visual Collaborative Perception [9.117534139771738]
  Multi-agent collaborative perception has emerged as a widely recognized technology in the field of autonomous driving. Current collaborative perception predominantly relies on LiDAR point clouds, with significantly less attention given to methods using camera images. This work proposes an instance-level fusion transformer for visual collaborative perception.
  arXiv  Detail & Related papers  (2024-07-13T11:38:15Z)
• Improving Bird's Eye View Semantic Segmentation by Task Decomposition [42.57351039508863]
  We decompose the original BEV segmentation task into two stages, namely BEV map reconstruction and RGB-BEV feature alignment. Our approach simplifies the complexity of combining perception and generation into distinct steps, equipping the model to handle intricate and challenging scenes effectively.
  arXiv  Detail & Related papers  (2024-04-02T13:19:45Z)
• DA-BEV: Unsupervised Domain Adaptation for Bird's Eye View Perception [104.87876441265593]
  Camera-only Bird's Eye View (BEV) has demonstrated great potential in environment perception in a 3D space. Unsupervised domain adaptive BEV, which effective learning from various unlabelled target data, is far under-explored. We design DA-BEV, the first domain adaptive camera-only BEV framework that addresses domain adaptive BEV challenges by exploiting the complementary nature of image-view features and BEV features.
  arXiv  Detail & Related papers  (2024-01-13T04:21:24Z)
• EVP: Enhanced Visual Perception using Inverse Multi-Attentive Feature Refinement and Regularized Image-Text Alignment [40.328294121805456]
  This work builds on the previous work VPD which paved the way to use the Stable Diffusion network for computer vision tasks. We develop the Inverse Multi-Attentive Feature Refinement (IMAFR) module which enhances feature learning capabilities. Second, we propose a novel image-text alignment module for improved feature extraction of the Stable Diffusion backbone.
  arXiv  Detail & Related papers  (2023-12-13T22:20:45Z)
• Leveraging BEV Representation for 360-degree Visual Place Recognition [14.497501941931759]
  This paper investigates the advantages of using Bird's Eye View representation in 360-degree visual place recognition (VPR) We propose a novel network architecture that utilizes the BEV representation in feature extraction, feature aggregation, and vision-LiDAR fusion. The proposed BEV-based method is evaluated in ablation and comparative studies on two datasets.
  arXiv  Detail & Related papers  (2023-05-23T08:29:42Z)
• BEVPlace: Learning LiDAR-based Place Recognition using Bird's Eye View Images [20.30997801125592]
  We explore the potential of a different representation in place recognition, i.e. bird's eye view (BEV) images. A simple VGGNet trained on BEV images achieves comparable performance with the state-of-the-art place recognition methods in scenes of slight viewpoint changes. We develop a method to estimate the position of the query cloud, extending the usage of place recognition.
  arXiv  Detail & Related papers  (2023-02-28T05:37:45Z)
• StructVPR: Distill Structural Knowledge with Weighting Samples for Visual Place Recognition [49.58170209388029]
  Visual place recognition (VPR) is usually considered as a specific image retrieval problem. We propose StructVPR, a novel training architecture for VPR, to enhance structural knowledge in RGB global features. Ours achieves state-of-the-art performance while maintaining a low computational cost.
  arXiv  Detail & Related papers  (2022-12-02T02:52:01Z)
• Delving into the Devils of Bird's-eye-view Perception: A Review, Evaluation and Recipe [115.31507979199564]
  Learning powerful representations in bird's-eye-view (BEV) for perception tasks is trending and drawing extensive attention both from industry and academia. As sensor configurations get more complex, integrating multi-source information from different sensors and representing features in a unified view come of vital importance. The core problems for BEV perception lie in (a) how to reconstruct the lost 3D information via view transformation from perspective view to BEV; (b) how to acquire ground truth annotations in BEV grid; and (d) how to adapt and generalize algorithms as sensor configurations vary across different scenarios.
  arXiv  Detail & Related papers  (2022-09-12T15:29:13Z)
• M^2BEV: Multi-Camera Joint 3D Detection and Segmentation with Unified Birds-Eye View Representation [145.6041893646006]
  M$2$BEV is a unified framework that jointly performs 3D object detection and map segmentation. M$2$BEV infers both tasks with a unified model and improves efficiency.
  arXiv  Detail & Related papers  (2022-04-11T13:43:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.