Meanings and Measurements: Multi-Agent Probabilistic Grounding for Vision-Language Navigation

• URL: http://arxiv.org/abs/2603.19166v1
• Date: Thu, 19 Mar 2026 17:20:56 GMT
• Title: Meanings and Measurements: Multi-Agent Probabilistic Grounding for Vision-Language Navigation
• Abstract summary: We show that vision language models (VLMs) are not explicitly designed to reason about metric constraints in physically defined spaces.<n>We propose MAPG, an agentic framework that decomposes language queries into structured subcomponents and queries a VLM to ground each component.<n> MAPG then probabilistically composes these grounded outputs to produce metrically consistent, actionable decisions in 3D space.
• Score: 9.838782060508109
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Robots collaborating with humans must convert natural language goals into actionable, physically grounded decisions. For example, executing a command such as "go two meters to the right of the fridge" requires grounding semantic references, spatial relations, and metric constraints within a 3D scene. While recent vision language models (VLMs) demonstrate strong semantic grounding capabilities, they are not explicitly designed to reason about metric constraints in physically defined spaces. In this work, we empirically demonstrate that state-of-the-art VLM-based grounding approaches struggle with complex metric-semantic language queries. To address this limitation, we propose MAPG (Multi-Agent Probabilistic Grounding), an agentic framework that decomposes language queries into structured subcomponents and queries a VLM to ground each component. MAPG then probabilistically composes these grounded outputs to produce metrically consistent, actionable decisions in 3D space. We evaluate MAPG on the HM-EQA benchmark and show consistent performance improvements over strong baselines. Furthermore, we introduce a new benchmark, MAPG-Bench, specifically designed to evaluate metric-semantic goal grounding, addressing a gap in existing language grounding evaluations. We also present a real-world robot demonstration showing that MAPG transfers beyond simulation when a structured scene representation is available.

Related papers

• ReSiReg: Towards Spatially Consistent Semantics in Language-Conditioned Robotic Tasks [38.74020024588315]
  We propose ReSiReg, a feature reconstruction method that uses spatially consistent VLM intermediates to improve dense language-grounded retrieval.<n>We evaluate quantitatively on OVSS and 3D mapping across backbones, and qualitatively in real-world manipulation scenes.<n>We provide a compact 25M dense VLM for robotic applications, substantially smaller than and competitive with ViT-B baselines.
  arXiv  Detail & Related papers  (2026-06-17T13:58:06Z)
• Geometric Action Model for Robot Policy Learning [68.6657929619782]
  Generalist robot policies must follow user instructions while reasoning about how objects, cameras, and robot actions interact in the 3D physical world.<n>Recent vision-language-action models (VLAs) and video world-action models (WAMs) inherit strong semantic or temporal priors from large-scale foundation models.<n>We propose the Geometric Action Model (GAM), a language-conditioned manipulation policy.
  arXiv  Detail & Related papers  (2026-06-15T17:58:03Z)
• Language as a Sensor: Calibrated Spatial Belief Estimation in 3D Scenes from Natural Language [15.05613635418399]
  Traditional metric-semantic mapping ignores language descriptions of parts of the world beyond their perceptual field of view.<n>We train a Language Sensor Model (LSM) that maps each utterance and its scene-graph context to a multimodal distribution.<n>We then introduce VL-Map (Vision-Language Metric-Semantic Mapping), a framework that treats these language predictions as observations and fuses them with onboard perception.
  arXiv  Detail & Related papers  (2026-06-07T15:17:25Z)
• EmbodimentSemantic: A Spatial Scene-Graph Dataset and Benchmark for Vision-Language Models on Embodied Manipulation Trajectories [16.996900088118412]
  We introduce EmbodimentSemantic, a spatial scene-graph dataset and benchmark for evaluating relational grounding in embodied manipulation.<n>The dataset includes real-world manipulation observations collected with the low-cost SO101 robot arm.<n>To provide controlled validation, we also introduce a simulator-grounded LIBERO benchmark with over 60K manipulation frames and more than 120K camera-specific scene graphs across paired third-person and wrist views.
  arXiv  Detail & Related papers  (2026-06-06T18:58:54Z)
• AffordanceVLA: A Vision-Language-Action Model Empowering Action Generation through Affordance-Aware Understanding [55.535374902204296]
  AffordanceVLA is a unified framework that introduces structured affordance forecasting as a task-oriented intermediate representation.<n>We show that AffordanceVLA achieves strong performance across diverse manipulation scenarios.
  arXiv  Detail & Related papers  (2026-06-04T13:28:51Z)
• Relational Scene Graphs for Object Grounding of Natural Language Commands [3.873578776565377]
  This paper investigates whether incorporating open- or closed-vocabulary spatial relations into 3DSGs can improve the ability of Large language models to interpret natural language commands.<n>To address this, we propose an LLM-based pipeline for target object grounding from open-vocabulary language commands and a vision language model (VLM)-based pipeline to add open-vocabulary spatial edges to 3DSGs from images captured while mapping.
  arXiv  Detail & Related papers  (2026-02-04T15:05:29Z)
• From Indoor to Open World: Revealing the Spatial Reasoning Gap in MLLMs [65.04549036809557]
  We introduce a benchmark built from pedestrian-perspective videos captured with synchronized stereo cameras, LiDAR, and IMU/GPS sensors.<n>This dataset provides metrically precise 3D information, enabling the automatic generation of spatial reasoning questions.<n> Evaluations reveal that the performance gains observed in structured indoor benchmarks vanish in open-world settings.
  arXiv  Detail & Related papers  (2025-12-22T18:58:12Z)
• Words into World: A Task-Adaptive Agent for Language-Guided Spatial Retrieval in AR [8.295391485284298]
  We present a modular augmented reality (AR) agent system that integrates multimodal large language models (MLLMs) with grounded vision models.<n>Our adaptive task agent coordinates MLLMs and coordinate-aware perception tools to address varying query complexities.<n>The system guides human attention to information-dense areas while supporting human-in-the-loop refinement.
  arXiv  Detail & Related papers  (2025-11-29T03:29:15Z)
• Executable Analytic Concepts as the Missing Link Between VLM Insight and Precise Manipulation [70.8381970762877]
  Vision-Language Models (VLMs) have demonstrated remarkable capabilities in semantic reasoning and task planning.<n>We introduce GRACE, a novel framework that grounds VLM-based reasoning through executable analytic concepts.<n>G GRACE provides a unified and interpretable interface between high-level instruction understanding and low-level robot control.
  arXiv  Detail & Related papers  (2025-10-09T09:08:33Z)
• ReasonGrounder: LVLM-Guided Hierarchical Feature Splatting for Open-Vocabulary 3D Visual Grounding and Reasoning [68.4209681278336]
  Open-vocabulary 3D visual grounding and reasoning aim to localize objects in a scene based on implicit language descriptions.<n>Current methods struggle because they rely heavily on fine-tuning with 3D annotations and mask proposals.<n>We propose ReasonGrounder, an LVLM-guided framework that uses hierarchical 3D feature Gaussian fields for adaptive grouping.
  arXiv  Detail & Related papers  (2025-03-30T03:40:35Z)
• IRef-VLA: A Benchmark for Interactive Referential Grounding with Imperfect Language in 3D Scenes [10.139461308573336]
  IRef-VLA is the largest real-world dataset for the referential grounding task consisting of over 11.5K scanned 3D rooms.<n>We aim to provide a resource for 3D scene understanding that aids the development of robust, interactive navigation systems.
  arXiv  Detail & Related papers  (2025-03-20T16:16:10Z)
• Towards Open-World Grasping with Large Vision-Language Models [5.317624228510749]
  An open-world grasping system should be able to combine high-level contextual with low-level physical-geometric reasoning. We propose OWG, an open-world grasping pipeline that combines vision-language models with segmentation and grasp synthesis models. We conduct evaluation in cluttered indoor scene datasets to showcase OWG's robustness in grounding from open-ended language.
  arXiv  Detail & Related papers  (2024-06-26T19:42:08Z)
• Grounding Language Plans in Demonstrations Through Counterfactual Perturbations [25.19071357445557]
  Grounding the common-sense reasoning of Large Language Models (LLMs) in physical domains remains a pivotal yet unsolved problem for embodied AI. We show our approach improves the interpretability and reactivity of imitation learning through 2D navigation and simulated and real robot manipulation tasks.
  arXiv  Detail & Related papers  (2024-03-25T19:04:59Z)
• Grounded Decoding: Guiding Text Generation with Grounded Models for Embodied Agents [111.15288256221764]
  Grounded-decoding project aims to solve complex, long-horizon tasks in a robotic setting by leveraging the knowledge of both models. We frame this as a problem similar to probabilistic filtering: decode a sequence that both has high probability under the language model and high probability under a set of grounded model objectives. We demonstrate how such grounded models can be obtained across three simulation and real-world domains, and that the proposed decoding strategy is able to solve complex, long-horizon tasks in a robotic setting by leveraging the knowledge of both models.
  arXiv  Detail & Related papers  (2023-03-01T22:58:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.