Loc3R-VLM: Language-based Localization and 3D Reasoning with Vision-Language Models

• URL: http://arxiv.org/abs/2603.18002v1
• Date: Wed, 18 Mar 2026 17:59:10 GMT
• Title: Loc3R-VLM: Language-based Localization and 3D Reasoning with Vision-Language Models
• Abstract summary: Loc3R-VLM is a framework that equips 2D Vision-Language Models with advanced 3D understanding capabilities from monocular video input.<n>Inspired by human spatial cognition, Loc3R-VLM relies on two joint objectives: global layout reconstruction and explicit situation modeling.<n>To ensure geometric consistency and metric-scale alignment, we leverage lightweight camera pose priors extracted from a pre-trained 3D foundation model.
• Score: 47.045362895601556
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal Large Language Models (MLLMs) have made impressive progress in connecting vision and language, but they still struggle with spatial understanding and viewpoint-aware reasoning. Recent efforts aim to augment the input representations with geometric cues rather than explicitly teaching models to reason in 3D space. We introduce Loc3R-VLM, a framework that equips 2D Vision-Language Models with advanced 3D understanding capabilities from monocular video input. Inspired by human spatial cognition, Loc3R-VLM relies on two joint objectives: global layout reconstruction to build a holistic representation of the scene structure, and explicit situation modeling to anchor egocentric perspective. These objectives provide direct spatial supervision that grounds both perception and language in a 3D context. To ensure geometric consistency and metric-scale alignment, we leverage lightweight camera pose priors extracted from a pre-trained 3D foundation model. Loc3R-VLM achieves state-of-the-art performance in language-based localization and outperforms existing 2D- and video-based approaches on situated and general 3D question-answering benchmarks, demonstrating that our spatial supervision framework enables strong 3D understanding. Project page: https://kevinqu7.github.io/loc3r-vlm

Related papers

• VLRC: Vision-Language Reprojection Consistency as a scalable signal for better feed-forward 3D pretraining [8.146936489399293]
  Vision-Language Reprojection Consistency (VLRC) exploits frozen vision-language representations as semantic multi-view supervision.<n>VLRC integrates seamlessly with both self-supervised monocular reconstruction and supervised-pretrained feed-forward 3D models.<n>Experiments on indoor and outdoor benchmarks demonstrate consistent gains in 3D reconstruction accuracy and zero-shot open-vocabulary 3D semantic segmentation.
  arXiv  Detail & Related papers  (2026-07-02T18:49:34Z)
• Occ-VLM: Occupancy Grounded Vision Language Model for Indoor Scene Understanding [31.76161781831759]
  We propose Occ-VLM, a novel framework for 3D scene understanding that operates purely on posed RGB images and employs a single 2D vision encoder.<n>Occ-VLM achieves both accurate geometric perception and robust vision-language reasoning.
  arXiv  Detail & Related papers  (2026-06-18T04:24:28Z)
• S$^2$-MLLM: Boosting Spatial Reasoning Capability of MLLMs for 3D Visual Grounding with Structural Guidance [20.55536735670125]
  3D Visual Grounding (3DVG) focuses on locating objects in 3D scenes based on natural language descriptions.<n>Recent advances in Multi-modal Large Language Models (MLLMs) have motivated research into extending them to 3DVG.<n>We propose S$2$-MLLM, an efficient framework that enhances spatial reasoning in MLLMs through implicit spatial reasoning.
  arXiv  Detail & Related papers  (2025-12-01T03:08:34Z)
• G$^2$VLM: Geometry Grounded Vision Language Model with Unified 3D Reconstruction and Spatial Reasoning [36.62798449863548]
  Vision-Language Models (VLMs) still lack robustness in spatial intelligence.<n>We present G$2$VLM, a vision-language model that bridges two fundamental aspects of spatial intelligence.
  arXiv  Detail & Related papers  (2025-11-26T18:59:39Z)
• Abstract 3D Perception for Spatial Intelligence in Vision-Language Models [100.13033631690114]
  Vision-language models (VLMs) struggle with 3D-related tasks such as spatial cognition and physical understanding.<n>We introduce SandboxVLM, a framework that leverages abstract bounding boxes to encode geometric structure and physical kinematics for VLM.<n>Our approach consistently improves spatial intelligence, achieving an 8.3% gain on SAT Real compared with baseline methods.
  arXiv  Detail & Related papers  (2025-11-14T04:16:09Z)
• GaussianVLM: Scene-centric 3D Vision-Language Models using Language-aligned Gaussian Splats for Embodied Reasoning and Beyond [56.677984098204696]
  multimodal language models are driving the development of 3D Vision-Language Models (VLMs)<n>We propose a scene-centric 3D VLM for 3D Gaussian splat scenes that employs language- and task-aware scene representations.<n>We present the first Gaussian splatting-based VLM, leveraging photorealistic 3D representations derived from standard RGB images.
  arXiv  Detail & Related papers  (2025-07-01T15:52:59Z)
• 3D-Aware Vision-Language Models Fine-Tuning with Geometric Distillation [17.294440057314812]
  Vision-Language Models (VLMs) have shown remarkable performance on diverse visual and linguistic tasks.<n>We propose Geometric Distillation, a framework that injects human-inspired geometric cues into pretrained VLMs.<n>Our method shapes representations to be geometry-aware while remaining compatible with natural image-text inputs.
  arXiv  Detail & Related papers  (2025-06-11T15:56:59Z)
• VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction [86.82819259860186]
  We introduce VLM-3R, a unified framework for Vision-Language Models (VLMs) that incorporates 3D Reconstructive instruction tuning.<n>VLM-3R processes monocular video frames by employing a geometry encoder to derive implicit 3D tokens that represent spatial understanding.
  arXiv  Detail & Related papers  (2025-05-26T17:56:30Z)
• Dynam3D: Dynamic Layered 3D Tokens Empower VLM for Vision-and-Language Navigation [61.21302433849139]
  Vision-and-Language Navigation (VLN) is a core task where embodied agents leverage their spatial mobility to navigate in 3D environments.<n>We propose Dynam3D, a dynamic layered 3D representation model that leverages language-aligned, generalizable, and hierarchical 3D representations as visual input to train 3D-VLM in navigation action prediction.<n>Our Dynam3D is capable of online encoding and localization of 3D instances, and dynamically updates them in changing environments to provide large-scale exploration and long-term memory capabilities for navigation.
  arXiv  Detail & Related papers  (2025-05-16T15:46:27Z)
• LanguageRefer: Spatial-Language Model for 3D Visual Grounding [72.7618059299306]
  We develop a spatial-language model for a 3D visual grounding problem. We show that our model performs competitively on visio-linguistic datasets proposed by ReferIt3D.
  arXiv  Detail & Related papers  (2021-07-07T18:55:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.