Leveraging Semantic Maps for City-Scale Cross-View Localization

• URL: http://arxiv.org/abs/2607.25215v1
• Date: Tue, 28 Jul 2026 02:44:56 GMT
• Title: Leveraging Semantic Maps for City-Scale Cross-View Localization
• Abstract summary: We want robots to localize in untraversed environments against commonly available prior data.<n>Existing methods either ignore this semantic information, directly matching panoramas and overhead imagery, or dramatically compress the semantic information.<n>We propose distilling a lightweight matcher from a VLM which computes correspondences for all entities in a map.
• Score: 10.247287191012036
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We want robots to localize in previously untraversed environments against commonly available prior data. Rich semantic data available from OpenStreetMap can be useful in this task. However, existing methods either ignore this semantic information, directly matching panoramas and overhead imagery, or dramatically compress the semantic information, working with a small set of fixed classes. To leverage this rich semantic information, two challenges need to be overcome. First, useful semantic information needs to be extracted from the robot's egocentric observations. Second, the observed information must be quickly associated with the large prior semantic map (e.g., up to 628 km^2). We show that VLMs are effective at both extracting relevant landmarks from panoramas, and identifying feasible correspondences between these landmarks and prior overhead landmarks. However, using VLMs to propose all correspondences scales poorly as the number of mapped landmarks increases. Instead, we propose distilling a lightweight matcher from a VLM which computes correspondences for all entities in a map. We use this output to form an observation likelihood which is fused over time with a Bayes filter to create a time series of pose estimates. To support further investigation into generalizable cross-view methods that leverage semantic information, we release a dataset of extracted semantics and evaluation trajectories spanning eleven environments, including panoramas we collected in a snowstorm and at night in Boston. We demonstrate our method, trained on a single city's fair-weather data, generalizes across location, lighting, weather, and other challenges. Code and datasets are available at https://efahnestock.github.io/loci/.

Related papers

• From Street to Orbit: Training-Free Cross-View Retrieval via Location Semantics and LLM Guidance [10.533095161205358]
  Cross-view image retrieval is critical for applications such as autonomous navigation, urban planning, and localization in GPS-denied environments.<n>We present a simple yet effective cross-view image retrieval framework that leverages a pretrained vision encoder and a large language model (LLM)<n>Despite using no ground-truth supervision or finetuning, our proposed method outperforms prior learning-based approaches on the benchmark dataset under zero-shot settings.
  arXiv  Detail & Related papers  (2025-11-12T23:51:46Z)
• Bayesian Fields: Task-driven Open-Set Semantic Gaussian Splatting [19.187033477078177]
  Open-set semantic mapping requires (i) determining the correct granularity to represent the scene.<n>(ii) fusing semantic knowledge across multiple 2D observations into an overall 3D reconstruction.<n>Bayesian Fields is a task-driven and probabilistic approach for open-set semantic mapping.
  arXiv  Detail & Related papers  (2025-03-07T21:36:30Z)
• GOTPR: General Outdoor Text-based Place Recognition Using Scene Graph Retrieval with OpenStreetMap [4.51019574688293]
  We propose GOTPR, a robust place recognition method designed for outdoor environments where GPS signals are unavailable.<n>Unlike existing approaches that use point cloud maps, which are large and difficult to store, GOTPR leverages scene graphs generated from text descriptions and maps for place recognition.<n>In city-scale tests, it completed processing within a few seconds, making it highly practical for real-world robotics applications.
  arXiv  Detail & Related papers  (2025-01-15T04:51:10Z)
• Efficient Masked AutoEncoder for Video Object Counting and A Large-Scale Benchmark [52.339936954958034]
  The dynamic imbalance of the fore-background is a major challenge in video object counting.<n>We propose a density-embedded Efficient Masked Autoencoder Counting (E-MAC) framework in this paper.<n>In addition, we first propose a large video bird counting dataset, DroneBird, in natural scenarios for migratory bird protection.
  arXiv  Detail & Related papers  (2024-11-20T06:08:21Z)
• SEER-ZSL: Semantic Encoder-Enhanced Representations for Generalized Zero-Shot Learning [0.6792605600335813]
  Zero-Shot Learning (ZSL) presents the challenge of identifying categories not seen during training.<n>We introduce a Semantic-Enhanced Representations for Zero-Shot Learning (SEER-ZSL)<n>First, we aim to distill meaningful semantic information using a probabilistic encoder, enhancing the semantic consistency and robustness.<n>Second, we distill the visual space by exploiting the learned data distribution through an adversarially trained generator. Third, we align the distilled information, enabling a mapping of unseen categories onto the true data manifold.
  arXiv  Detail & Related papers  (2023-12-20T15:18:51Z)
• Cross-attention Spatio-temporal Context Transformer for Semantic Segmentation of Historical Maps [18.016789471815855]
  Historical maps provide useful-temporal information on the Earth's surface before modern earth observation techniques came into being. Aleatoric uncertainty known as data-dependent uncertainty inherent in the drawing/fading defects of the original map sheets. We propose a U--based network that fuses maps that aggregating information at a larger range as well as through a temporal sequence.
  arXiv  Detail & Related papers  (2023-10-19T09:49:58Z)
• Neural Semantic Surface Maps [52.61017226479506]
  We present an automated technique for computing a map between two genus-zero shapes, which matches semantically corresponding regions to one another. Our approach can generate semantic surface-to-surface maps, eliminating manual annotations or any 3D training data requirement.
  arXiv  Detail & Related papers  (2023-09-09T16:21:56Z)
• Learning Implicit Feature Alignment Function for Semantic Segmentation [51.36809814890326]
  Implicit Feature Alignment function (IFA) is inspired by the rapidly expanding topic of implicit neural representations. We show that IFA implicitly aligns the feature maps at different levels and is capable of producing segmentation maps in arbitrary resolutions. Our method can be combined with improvement on various architectures, and it achieves state-of-the-art accuracy trade-off on common benchmarks.
  arXiv  Detail & Related papers  (2022-06-17T09:40:14Z)
• Data Augmentation for Object Detection via Differentiable Neural Rendering [71.00447761415388]
  It is challenging to train a robust object detector when annotated data is scarce. Existing approaches to tackle this problem include semi-supervised learning that interpolates labeled data from unlabeled data. We introduce an offline data augmentation method for object detection, which semantically interpolates the training data with novel views.
  arXiv  Detail & Related papers  (2021-03-04T06:31:06Z)
• Towards Semantic Segmentation of Urban-Scale 3D Point Clouds: A Dataset, Benchmarks and Challenges [52.624157840253204]
  We present an urban-scale photogrammetric point cloud dataset with nearly three billion richly annotated points. Our dataset consists of large areas from three UK cities, covering about 7.6 km2 of the city landscape. We evaluate the performance of state-of-the-art algorithms on our dataset and provide a comprehensive analysis of the results.
  arXiv  Detail & Related papers  (2020-09-07T14:47:07Z)
• Exploiting Multi-Layer Grid Maps for Surround-View Semantic Segmentation of Sparse LiDAR Data [2.6876976011647145]
  We consider the transformation of laser range measurements into a top-view grid map representation to approach the task of LiDAR-only semantic segmentation. We are exploiting a grid map framework to extract relevant information and represent them by using multi-layer grid maps. We compare single-layer and multi-layer approaches and demonstrate the benefit of a multi-layer grid map input.
  arXiv  Detail & Related papers  (2020-05-13T23:50:34Z)
• Weakly-Supervised Salient Object Detection via Scribble Annotations [54.40518383782725]
  We propose a weakly-supervised salient object detection model to learn saliency from scribble labels. We present a new metric, termed saliency structure measure, to measure the structure alignment of the predicted saliency maps. Our method not only outperforms existing weakly-supervised/unsupervised methods, but also is on par with several fully-supervised state-of-the-art models.
  arXiv  Detail & Related papers  (2020-03-17T12:59:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.