VLM-Loc: Localization in Point Cloud Maps via Vision-Language Models

• URL: http://arxiv.org/abs/2603.09826v1
• Date: Tue, 10 Mar 2026 15:48:25 GMT
• Title: VLM-Loc: Localization in Point Cloud Maps via Vision-Language Models
• Abstract summary: Text-to-point-cloud (T2P) localization aims to infer precise spatial positions within 3D point cloud maps from natural language descriptions.<n>We propose VLM-Loc, a framework that leverages the spatial reasoning capability of large vision-language models.<n>Experiments on CityLoc demonstrate VLM-Loc achieves superior accuracy and robustness compared to state-of-the-art methods.
• Score: 33.748083718525756
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Text-to-point-cloud (T2P) localization aims to infer precise spatial positions within 3D point cloud maps from natural language descriptions, reflecting how humans perceive and communicate spatial layouts through language. However, existing methods largely rely on shallow text-point cloud correspondence without effective spatial reasoning, limiting their accuracy in complex environments. To address this limitation, we propose VLM-Loc, a framework that leverages the spatial reasoning capability of large vision-language models (VLMs) for T2P localization. Specifically, we transform point clouds into bird's-eye-view (BEV) images and scene graphs that jointly encode geometric and semantic context, providing structured inputs for the VLM to learn cross-modal representations bridging linguistic and spatial semantics. On top of these representations, we introduce a partial node assignment mechanism that explicitly associates textual cues with scene graph nodes, enabling interpretable spatial reasoning for accurate localization. To facilitate systematic evaluation across diverse scenes, we present CityLoc, a benchmark built from multi-source point clouds for fine-grained T2P localization. Experiments on CityLoc demonstrate VLM-Loc achieves superior accuracy and robustness compared to state-of-the-art methods. Our code, model, and dataset are available at \href{https://github.com/MCG-NKU/nku-3d-vision}{repository}.

Related papers

• PruneGround: Plug-and-play Spatial Pruning for 3D Visual Grounding [45.510721162264666]
  3D Visual Grounding aims to localize target objects in 3D scenes given natural language descriptions.<n>Existing approaches typically perform reasoning over the entire scene, leading to ambiguous predictions and high computational cost.<n>We propose PruneGround, an effective plug-and-play framework for 3DVG built upon three key components.
  arXiv  Detail & Related papers  (2026-06-30T05:21:11Z)
• Text2Loc++: Generalizing 3D Point Cloud Localization from Natural Language [44.7011717447999]
  We present Text2Loc++, a novel neural network designed for effective cross-modal alignment between language and point clouds.<n>To support benchmarking, we introduce a new city-scale dataset covering color and non-color point clouds from diverse urban scenes.<n>In the global place recognition stage, Text2Loc++ combines a pretrained language model with a Hierarchical Transformer with Max pooling (HTM) for sentence-level semantics.<n>In the fine localization stage, we completely remove explicit text-instance matching and design a lightweight yet powerful framework.
  arXiv  Detail & Related papers  (2025-11-19T10:19:45Z)
• A Neural Representation Framework with LLM-Driven Spatial Reasoning for Open-Vocabulary 3D Visual Grounding [78.99798110890157]
  Open-vocabulary 3D visual grounding aims to localize target objects based on free-form language queries.<n>Existing language field methods struggle to accurately localize instances using spatial relations in language queries.<n>We propose SpatialReasoner, a novel neural representation-based framework with large language model (LLM)-driven spatial reasoning.
  arXiv  Detail & Related papers  (2025-07-09T10:20:38Z)
• Agentic 3D Scene Generation with Spatially Contextualized VLMs [67.31920821192323]
  We introduce a new paradigm that enables vision-language models to generate, understand, and edit complex 3D environments.<n>We develop an agentic 3D scene generation pipeline in which the VLM iteratively reads from and updates the spatial context.<n>Results show that our framework can handle diverse and challenging inputs, achieving a level of generalization not observed in prior work.
  arXiv  Detail & Related papers  (2025-05-26T15:28:17Z)
• LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Models [57.92316645992816]
  Spatial reasoning is a fundamental aspect of human cognition, enabling intuitive understanding and manipulation of objects in three-dimensional space.<n>We introduce LayoutVLM, a framework and scene layout representation that exploits the semantic knowledge of Vision-Language Models (VLMs)<n>We demonstrate that fine-tuning VLMs with the proposed scene layout representation extracted from existing scene datasets can improve their reasoning performance.
  arXiv  Detail & Related papers  (2024-12-03T06:15:04Z)
• Talking to DINO: Bridging Self-Supervised Vision Backbones with Language for Open-Vocabulary Segmentation [56.001484215308075]
  We present Talk2DINO, a novel hybrid approach that combines the spatial accuracy of DINOv2 with the language understanding of CLIP.<n>We show that the powerful semantic and localization abilities of Talk2DINO can enhance the segmentation process.<n> Experimental results demonstrate that Talk2DINO achieves state-of-the-art performance across several unsupervised OVS benchmarks.
  arXiv  Detail & Related papers  (2024-11-28T19:00:03Z)
• SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models [68.13636352687257]
  We introduce Spatial Region GPT (SpatialRGPT) to enhance VLMs' spatial perception and reasoning capabilities. During inference, when provided with user-specified region proposals, SpatialRGPT can accurately perceive their relative directions and distances. Our results demonstrate that SpatialRGPT significantly enhances performance in spatial reasoning tasks, both with and without local region prompts.
  arXiv  Detail & Related papers  (2024-06-03T17:59:06Z)
• Instance-free Text to Point Cloud Localization with Relative Position Awareness [37.22900045434484]
  Text-to-point-cloud cross-modal localization is an emerging vision-language task critical for future robot-human collaboration. We address two key limitations of existing approaches: 1) their reliance on ground-truth instances as input; and 2) their neglect of the relative positions among potential instances. Our proposed model follows a two-stage pipeline, including a coarse stage for text-cell retrieval and a fine stage for position estimation.
  arXiv  Detail & Related papers  (2024-04-27T09:46:49Z)
• O2V-Mapping: Online Open-Vocabulary Mapping with Neural Implicit Representation [9.431926560072412]
  We propose O2V-mapping, which utilizes voxel-based language and geometric features to create an open-vocabulary field.<n>Experiments on open-vocabulary object localization and semantic segmentation demonstrate that O2V-mapping achieves online construction of language scenes.
  arXiv  Detail & Related papers  (2024-04-10T08:54:43Z)
• Text2Loc: 3D Point Cloud Localization from Natural Language [49.01851743372889]
  We tackle the problem of 3D point cloud localization based on a few natural linguistic descriptions. We introduce a novel neural network, Text2Loc, that fully interprets the semantic relationship between points and text. Text2Loc improves the localization accuracy by up to $2times$ over the state-of-the-art on the KITTI360Pose dataset.
  arXiv  Detail & Related papers  (2023-11-27T16:23:01Z)
• CPCM: Contextual Point Cloud Modeling for Weakly-supervised Point Cloud Semantic Segmentation [60.0893353960514]
  We study the task of weakly-supervised point cloud semantic segmentation with sparse annotations. We propose a Contextual Point Cloud Modeling ( CPCM) method that consists of two parts: a region-wise masking (RegionMask) strategy and a contextual masked training (CMT) method.
  arXiv  Detail & Related papers  (2023-07-19T04:41:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.