Language as a Sensor: Calibrated Spatial Belief Estimation in 3D Scenes from Natural Language

• URL: http://arxiv.org/abs/2606.08666v1
• Date: Sun, 07 Jun 2026 15:17:25 GMT
• Title: Language as a Sensor: Calibrated Spatial Belief Estimation in 3D Scenes from Natural Language
• Abstract summary: Traditional metric-semantic mapping ignores language descriptions of parts of the world beyond their perceptual field of view.<n>We train a Language Sensor Model (LSM) that maps each utterance and its scene-graph context to a multimodal distribution.<n>We then introduce VL-Map (Vision-Language Metric-Semantic Mapping), a framework that treats these language predictions as observations and fuses them with onboard perception.
• Score: 15.05613635418399
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Robots deployed in human-centric environments routinely receive natural-language descriptions of spatial information ("I left my backpack on the table") that reference parts of the world beyond their perceptual field of view. Traditional metric-semantic mapping ignores this signal, while off-the-shelf multimodal models remain limited in 3D spatial reasoning and are not directly amenable to fusion with other sensor modalities. To convert language observations into a calibrated spatial distribution, we train a Language Sensor Model (LSM) that maps each utterance and its scene-graph context to a multimodal distribution, with mixture weights encoding referential ambiguity (e.g., "which table") and component covariances encoding spatial uncertainty (e.g., where "on the table" the target lies). We then introduce VL-Map (Vision-Language Metric-Semantic Mapping), a probabilistic framework that treats these language predictions as stochastic observations and fuses them with onboard perception within a unified belief map. On the VLA-3D benchmark as well as on a real-world mobile robot, LSM is the only language predictor whose covariance estimates remain within the calibrated regime; fused into VL-Map, it leads to more accurate predictions of the target object location (~70% more probability mass on the true target compared to the strongest foundation-model baseline).

Related papers

• VEGA: Visual Encoder Grounding Alignment for Spatially-Aware Vision-Language-Action Models [55.12929235609365]
  Visual backbones of current vision-language-action (VLA) models are predominantly pretrained on 2D image data without explicit 3D geometric supervision.<n>Existing implicit grounding methods partially address this by aligning VLA features with those of 3D-aware foundation models.<n>We propose VEGA, a framework that directly aligns the output of the VLA's visual encoder with spatially-aware features from DINOv2-FiT3D.
  arXiv  Detail & Related papers  (2026-05-11T12:44:26Z)
• Scalable Object Relation Encoding for Better 3D Spatial Reasoning in Large Language Models [50.14156501544165]
  spatial reasoning focuses on locating target objects based on spatial relations in 3D scenes.<n>Previous approaches have attempted to inject 3D scene representations into the input space of Large Language Models.<n>We propose QuatRoPE, a novel positional embedding method with an input length that is linear to the number of objects.
  arXiv  Detail & Related papers  (2026-03-25T18:46:23Z)
• Meanings and Measurements: Multi-Agent Probabilistic Grounding for Vision-Language Navigation [9.838782060508109]
  We show that vision language models (VLMs) are not explicitly designed to reason about metric constraints in physically defined spaces.<n>We propose MAPG, an agentic framework that decomposes language queries into structured subcomponents and queries a VLM to ground each component.<n> MAPG then probabilistically composes these grounded outputs to produce metrically consistent, actionable decisions in 3D space.
  arXiv  Detail & Related papers  (2026-03-19T17:20:56Z)
• From Indoor to Open World: Revealing the Spatial Reasoning Gap in MLLMs [65.04549036809557]
  We introduce a benchmark built from pedestrian-perspective videos captured with synchronized stereo cameras, LiDAR, and IMU/GPS sensors.<n>This dataset provides metrically precise 3D information, enabling the automatic generation of spatial reasoning questions.<n> Evaluations reveal that the performance gains observed in structured indoor benchmarks vanish in open-world settings.
  arXiv  Detail & Related papers  (2025-12-22T18:58:12Z)
• VLG-Loc: Vision-Language Global Localization from Labeled Footprint Maps [5.851042749720534]
  A vision-language model (VLM) is used to search the robot's multi-directional image observations for the landmarks noted in the map.<n>The method then identifies robot poses within a Monte Carlo localization framework.<n> Experimental validation in simulated and real-world retail environments demonstrates superior robustness compared to existing scan-based methods.
  arXiv  Detail & Related papers  (2025-12-14T18:22:00Z)
• A Neural Representation Framework with LLM-Driven Spatial Reasoning for Open-Vocabulary 3D Visual Grounding [78.99798110890157]
  Open-vocabulary 3D visual grounding aims to localize target objects based on free-form language queries.<n>Existing language field methods struggle to accurately localize instances using spatial relations in language queries.<n>We propose SpatialReasoner, a novel neural representation-based framework with large language model (LLM)-driven spatial reasoning.
  arXiv  Detail & Related papers  (2025-07-09T10:20:38Z)
• Multimodal Spatial Language Maps for Robot Navigation and Manipulation [32.852583241593436]
  multimodal spatial language maps are a spatial map representation that fuses pretrained multimodal features with a 3D reconstruction of the environment.<n>We present two instances of our maps, which are visual-language maps (VLMaps) and their extension to audio-visual-language maps (AVLMaps)<n>These capabilities extend to mobile robots and tabletop manipulators, supporting navigation and interaction guided by visual, audio, and spatial cues.
  arXiv  Detail & Related papers  (2025-06-07T17:02:13Z)
• Mind the Gap: Benchmarking Spatial Reasoning in Vision-Language Models [14.442394137843923]
  We present a detailed analysis that first delineates the core elements of spatial reasoning.<n>We then assesses the performance of these models in both synthetic and real-world images.
  arXiv  Detail & Related papers  (2025-03-25T14:34:06Z)
• Volumetric Semantically Consistent 3D Panoptic Mapping [77.13446499924977]
  We introduce an online 2D-to-3D semantic instance mapping algorithm aimed at generating semantic 3D maps suitable for autonomous agents in unstructured environments. It introduces novel ways of integrating semantic prediction confidence during mapping, producing semantic and instance-consistent 3D regions. The proposed method achieves accuracy superior to the state of the art on public large-scale datasets, improving on a number of widely used metrics.
  arXiv  Detail & Related papers  (2023-09-26T08:03:10Z)
• Kimera-Multi: Robust, Distributed, Dense Metric-Semantic SLAM for Multi-Robot Systems [92.26462290867963]
  Kimera-Multi is the first multi-robot system that is robust and capable of identifying and rejecting incorrect inter and intra-robot loop closures. We demonstrate Kimera-Multi in photo-realistic simulations, SLAM benchmarking datasets, and challenging outdoor datasets collected using ground robots.
  arXiv  Detail & Related papers  (2021-06-28T03:56:40Z)
• Spatial Language Understanding for Object Search in Partially Observed Cityscale Environments [21.528770932332474]
  We introduce the spatial language observation space and formulate a model under the framework of Partially Observable Markov Decision Process (POMDP) We propose a convolutional neural network model that learns to predict the language provider's relative frame of reference (FoR) given environment context. We demonstrate the generalizability of our FoR prediction model and object search system through cross-validation over areas of five cities, each with a 40,000m$2$ footprint.
  arXiv  Detail & Related papers  (2020-12-04T16:27:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.