Where Am I? Semantic Map Grounding via Vision-Language Models for Multi-Modal Localization

• URL: http://arxiv.org/abs/2607.01079v1
• Date: Wed, 01 Jul 2026 15:37:29 GMT
• Title: Where Am I? Semantic Map Grounding via Vision-Language Models for Multi-Modal Localization
• Abstract summary: We address robot localization in GPS-denied indoor environments by reframing it as a semantic reasoning task.<n>Motivated by how humans localize using object-level cues and labeled maps, we ask whether a vision-language model, given a front camera image, can infer the robot pose.<n>We fine-tune Qwen2.5-VL-7B with LoRA and attach a lightweight regression head that predicts continuous pose coordinates.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We address robot localization in GPS-denied indoor environments by reframing it as a semantic reasoning task rather than a geometric estimation problem. Motivated by how humans localize using object-level cues and labeled maps, we ask whether a vision-language model, given a front camera image, a polar LiDAR scan, and a top-down semantic grid map, can infer the robot pose. We fine-tune Qwen2.5-VL-7B with LoRA and attach a lightweight regression head that predicts continuous pose coordinates (x, y, theta) directly from the final hidden state, bypassing text generation. Training uses a composite position-and-direction loss with curriculum learning on a custom Gazebo dataset of 120,112 samples and 527 scenes. On the in-distribution test set of 18,017 samples, the model achieves 98.23 percent position accuracy, 98.00 percent direction accuracy, 96.75 percent full pose accuracy, a mean position error of 0.11 m, and a mean orientation error of 5.7 degrees at 0.62 s per sample. Position accuracy drops by only 7.2 percentage points on seven unseen object categories, reaching 90.99 percent, supporting semantic spatial reasoning rather than appearance memorization. With incomplete maps, fine-tuning recovers performance to 93.72 percent position accuracy, showing adaptability to stale or partial map information. Two ablations highlight cross-modal complementarity. Without LiDAR, using only camera and map inputs, position accuracy remains 95.06 percent, only 3.2 percentage points below the full system. However, when the camera sees no visible objects in a wall-facing view, LiDAR sustains 92.33 percent position accuracy, compared with 70.74 percent when neither LiDAR nor visible objects are available. This shows that LiDAR becomes the primary localization signal when camera semantics are unavailable and provides a reliable fallback under occlusion or sparse layouts.

Related papers

• Decodable Is Not Grounded: A Vision-Ablation Arbiter for VLM Spatial Reasoning [48.438568700670835]
  The standard way to read latent knowledge out of a model can systematically overstate what a vision-language model grounds in the image.<n>We show this on spatial reasoning, where the error is invisible to both probing and steering yet exposed by a one-line causal control.
  arXiv  Detail & Related papers  (2026-06-30T07:33:18Z)
• Efficient Visual Pointing for Embodied AI:Agent-Driven Data Synthesis, Cross-Block Attention, and Iterative Correction [55.11480729304395]
  We describe our PointArena 2026 solution, which achieves 77.2% overall accuracy and ranks second on the benchmark.<n>The ap proach targets three failure modes. First, agent-driven syn thesis builds large semantic and anchor-relative candidate pools.<n>Second, a determinis tic steerable-data pipeline creates a verified 10,000-sample main set, plus reserve samples, using masks, templates, and path verification.
  arXiv  Detail & Related papers  (2026-06-29T06:39:03Z)
• AnyImageNav: Any-View Geometry for Precise Last-Meter Image-Goal Navigation [10.175661009911257]
  We introduce AnyImageNav, a training-free system that pushes ImageNav toward a more demanding setting.<n>Our key insight is that the goal image can be treated as a geometric query.<n>Our method sets state-of-the-art navigation success rates on Gibson (93.1%) and HM3D (82.6%)
  arXiv  Detail & Related papers  (2026-04-07T02:45:07Z)
• Where on Earth? A Vision-Language Benchmark for Probing Model Geolocation Skills Across Scales [61.03549470159347]
  Vision-language models (VLMs) have advanced rapidly, yet their capacity for image-grounded geolocation in open-world conditions has not been comprehensively evaluated.<n>We present EarthWhere, a comprehensive benchmark for VLM image geolocation that evaluates visual recognition, step-by-step reasoning, and evidence use.
  arXiv  Detail & Related papers  (2025-10-13T01:12:21Z)
• Loc$^2$: Interpretable Cross-View Localization via Depth-Lifted Local Feature Matching [80.57282092735991]
  We propose an accurate and interpretable fine-grained cross-view localization method.<n>It estimates the 3 Degrees of Freedom (DoF) pose of a ground-level image by matching its local features with a reference aerial image.<n> Experiments show state-of-the-art accuracy in challenging scenarios such as cross-area testing and unknown orientation.
  arXiv  Detail & Related papers  (2025-09-11T18:52:16Z)
• Object-Pose Estimation With Neural Population Codes [4.557963624437784]
  Object symmetry complicates the direct mapping of sensory input to object rotation.<n>We show that representing object rotation with a neural population code overcomes these limitations.<n>We achieve inference in 3.2 milliseconds on an Apple M1 CPU.
  arXiv  Detail & Related papers  (2025-02-19T03:23:43Z)
• Telling Left from Right: Identifying Geometry-Aware Semantic Correspondence [80.6840060272386]
  This paper identifies the importance of being geometry-aware for semantic correspondence. We show that incorporating this information can markedly enhance semantic correspondence performance. Our method achieves a PCK@0.10 score of 65.4 (zero-shot) and 85.6 (supervised) on the challenging SPair-71k dataset.
  arXiv  Detail & Related papers  (2023-11-28T18:45:13Z)
• Convolutional Cross-View Pose Estimation [9.599356978682108]
  We propose a novel end-to-end method for cross-view pose estimation. Our method is validated on the VIGOR and KITTI datasets. On the Oxford RobotCar dataset, our method can reliably estimate the ego-vehicle's pose over time.
  arXiv  Detail & Related papers  (2023-03-09T13:52:28Z)
• Visual Cross-View Metric Localization with Dense Uncertainty Estimates [11.76638109321532]
  This work addresses visual cross-view metric localization for outdoor robotics. Given a ground-level color image and a satellite patch that contains the local surroundings, the task is to identify the location of the ground camera within the satellite patch. We devise a novel network architecture with denser satellite descriptors, similarity matching at the bottleneck, and a dense spatial distribution as output to capture multi-modal localization ambiguities.
  arXiv  Detail & Related papers  (2022-08-17T20:12:23Z)
• Beyond Cross-view Image Retrieval: Highly Accurate Vehicle Localization Using Satellite Image [91.29546868637911]
  This paper addresses the problem of vehicle-mounted camera localization by matching a ground-level image with an overhead-view satellite map. The key idea is to formulate the task as pose estimation and solve it by neural-net based optimization. Experiments on standard autonomous vehicle localization datasets have confirmed the superiority of the proposed method.
  arXiv  Detail & Related papers  (2022-04-10T19:16:58Z)
• Uncertainty-Aware Camera Pose Estimation from Points and Lines [101.03675842534415]
  Perspective-n-Point-and-Line (Pn$PL) aims at fast, accurate and robust camera localizations with respect to a 3D model from 2D-3D feature coordinates.
  arXiv  Detail & Related papers  (2021-07-08T15:19:36Z)
• 3D Lidar Mapping Relative Accuracy Automatic Evaluation Algorithm [7.4977759910523964]
  HD (High Definition) map based on 3D lidar plays a vital role in autonomous vehicle localization, planning, decision-making, perception, etc. Many 3D lidar mapping technologies related to SLAM are used in HD map construction to ensure its high accuracy. To evaluate the accuracy of 3D lidar mapping, the most common methods use ground truth of poses to calculate the error between estimated poses and ground truth. In this paper, we proposed a relative accuracy evaluation algorithm that can automatically evaluate the accuracy of HD map built by 3D lidar mapping without ground truth.
  arXiv  Detail & Related papers  (2020-06-01T11:30:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.