NavBench: Probing Multimodal Large Language Models for Embodied Navigation

• URL: http://arxiv.org/abs/2506.01031v1
• Date: Sun, 01 Jun 2025 14:21:02 GMT
• Title: NavBench: Probing Multimodal Large Language Models for Embodied Navigation
• Abstract summary: NavBench is a benchmark to evaluate the embodied navigation capabilities of MLLMs under zero-shot settings.<n>To support real-world deployment, we introduce a pipeline that converts MLLMs' outputs into robotic actions.
• Score: 21.787627598744358
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal Large Language Models (MLLMs) have demonstrated strong generalization in vision-language tasks, yet their ability to understand and act within embodied environments remains underexplored. We present NavBench, a benchmark to evaluate the embodied navigation capabilities of MLLMs under zero-shot settings. NavBench consists of two components: (1) navigation comprehension, assessed through three cognitively grounded tasks including global instruction alignment, temporal progress estimation, and local observation-action reasoning, covering 3,200 question-answer pairs; and (2) step-by-step execution in 432 episodes across 72 indoor scenes, stratified by spatial, cognitive, and execution complexity. To support real-world deployment, we introduce a pipeline that converts MLLMs' outputs into robotic actions. We evaluate both proprietary and open-source models, finding that GPT-4o performs well across tasks, while lighter open-source models succeed in simpler cases. Results also show that models with higher comprehension scores tend to achieve better execution performance. Providing map-based context improves decision accuracy, especially in medium-difficulty scenarios. However, most models struggle with temporal understanding, particularly in estimating progress during navigation, which may pose a key challenge.

Related papers

• Instance-Enriched Semantic Maps for Visual Language Navigation [9.24074870738429]
  Visual Language Navigation aims to enable an embodied agent to navigate complex environments by following natural language instructions.<n>Existing systems lack instance-level object detail and robustness to diverse user queries, limiting reliable navigation in complex indoor environments.<n>We propose Instance-Enriched Semantic Maps, a unified framework with three key contributions.
  arXiv  Detail & Related papers  (2026-07-14T11:09:48Z)
• Benchmarking Interaction, Beyond Policy: a Reproducible Benchmark for Collaborative Instance Object Navigation [54.6417539730783]
  Question-Asking Navigation (QAsk-Nav) is the first reproducible benchmark for Collaborative Instance Object Navigation (CoIN)<n>We develop Light-CoNav, a lightweight unified model for collaborative navigation that is 3x smaller and 70x faster than existing modular methods.
  arXiv  Detail & Related papers  (2026-03-31T21:45:00Z)
• SysNav: Multi-Level Systematic Cooperation Enables Real-World, Cross-Embodiment Object Navigation [46.34939555586507]
  We introduce SysNav, a three-level ObjectNav system for real-world crossembodiment deployment.<n> SysNav decouples semantic reasoning, navigation planning and motion control to ensure robustness and generalizability.<n>Our system achieves substantial improvements in both success rate and navigation efficiency.
  arXiv  Detail & Related papers  (2026-03-06T22:20:51Z)
• One Agent to Guide Them All: Empowering MLLMs for Vision-and-Language Navigation via Explicit World Representation [18.529673835965745]
  A navigable agent needs to understand both high-level semantic instructions and precise spatial perceptions.<n>We propose a decoupled design that separates low-level spatial state estimation from high-level semantic planning.<n>We conduct comprehensive experiments in both simulated and real-world environments.
  arXiv  Detail & Related papers  (2026-02-17T07:13:48Z)
• City Navigation in the Wild: Exploring Emergent Navigation from Web-Scale Knowledge in MLLMs [13.863236619171174]
  Task is designed to evaluate sequential decision-making abilities of MLLMs in challenging, knowledge-intensive real-world environments.<n>We operationalize this task with CityNav, a benchmark encompassing four diverse global cities.<n>Agents are required to rely solely on visual inputs and internal multimodal reasoning to sequentially navigate 50+ decision points.<n>We propose Verbalization of Path (VoP), which explicitly grounds the agent's internal reasoning by probing an explicit cognitive map.
  arXiv  Detail & Related papers  (2025-12-17T19:59:31Z)
• Actial: Activate Spatial Reasoning Ability of Multimodal Large Language Models [75.45940282834327]
  We introduce Viewpoint Learning, a task designed to evaluate and improve the spatial reasoning capabilities of MLLMs.<n>We present the Viewpoint-100K dataset, consisting of 100K object-centric image pairs with diverse viewpoints and corresponding question-answer pairs.<n>Our approach employs a two-stage fine-tuning strategy, resulting in significant improvements across multiple tasks.
  arXiv  Detail & Related papers  (2025-11-03T14:27:00Z)
• PointArena: Probing Multimodal Grounding Through Language-Guided Pointing [79.80132157576978]
  Pointing serves as a fundamental and intuitive mechanism for grounding language within visual contexts.<n>We introduce PointArena, a comprehensive platform for evaluating multimodal pointing across diverse reasoning scenarios.
  arXiv  Detail & Related papers  (2025-05-15T06:04:42Z)
• VOILA: Evaluation of MLLMs For Perceptual Understanding and Analogical Reasoning [63.0285363282581]
  Multimodal Large Language Models (MLLMs) have become a powerful tool for integrating visual and textual information.<n>We introduce VOILA, a benchmark designed to evaluate MLLMs' perceptual understanding and abstract relational reasoning.<n>We reveal that current MLLMs struggle to comprehend inter-image relationships and exhibit limited capabilities in high-level relational reasoning.
  arXiv  Detail & Related papers  (2025-02-25T23:36:19Z)
• EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents [63.43699771428243]
  EmbodiedBench is an extensive benchmark designed to evaluate vision-driven embodied agents.<n>We evaluated 19 leading proprietary and open-source MLLMs within EmbodiedBench.<n> MLLMs excel at high-level tasks but struggle with low-level manipulation.
  arXiv  Detail & Related papers  (2025-02-13T18:11:34Z)
• Uni-NaVid: A Video-based Vision-Language-Action Model for Unifying Embodied Navigation Tasks [24.690910258151693]
  Existing models for embodied navigation fall short of serving as practical generalists in the real world.<n>We present Uni-NaVid, the first video-based vision-language-action model designed to unify diverse embodied navigation tasks.<n>Uni-NaVid achieves this by the input and output data configurations for all commonly used embodied navigation tasks.
  arXiv  Detail & Related papers  (2024-12-09T05:55:55Z)
• DivScene: Towards Open-Vocabulary Object Navigation with Large Vision Language Models in Diverse Scenes [76.24687327731031]
  We first study the challenge of open-vocabulary object navigation by introducing DivScene.<n>Our dataset provides a much greater diversity of target objects and scene types than existing datasets.<n>We fine-tuned LVLMs to predict the next action with CoT explanations.
  arXiv  Detail & Related papers  (2024-10-03T17:49:28Z)
• PLANRL: A Motion Planning and Imitation Learning Framework to Bootstrap Reinforcement Learning [13.564676246832544]
  We introduce PLANRL, a framework that chooses when the robot should use classical motion planning and when it should learn a policy. PLANRL switches between two modes of operation: reaching a waypoint using classical techniques when away from the objects and fine-grained manipulation control when about to interact with objects. We evaluate our approach across multiple challenging simulation environments and real-world tasks, demonstrating superior performance in terms of adaptability, efficiency, and generalization compared to existing methods.
  arXiv  Detail & Related papers  (2024-08-07T19:30:08Z)
• NavCoT: Boosting LLM-Based Vision-and-Language Navigation via Learning Disentangled Reasoning [97.88246428240872]
  Vision-and-Language Navigation (VLN), as a crucial research problem of Embodied AI, requires an embodied agent to navigate through complex 3D environments following natural language instructions.<n>Recent research has highlighted the promising capacity of large language models (LLMs) in VLN by improving navigational reasoning accuracy and interpretability.<n>This paper introduces a novel strategy called Navigational Chain-of-Thought (NavCoT), where we fulfill parameter-efficient in-domain training to enable self-guided navigational decision.
  arXiv  Detail & Related papers  (2024-03-12T07:27:02Z)
• SayNav: Grounding Large Language Models for Dynamic Planning to Navigation in New Environments [14.179677726976056]
  SayNav is a new approach that leverages human knowledge from Large Language Models (LLMs) for efficient generalization to complex navigation tasks. SayNav achieves state-of-the-art results and even outperforms an oracle based baseline with strong ground-truth assumptions by more than 8% in terms of success rate.
  arXiv  Detail & Related papers  (2023-09-08T02:24:37Z)
• CorNav: Autonomous Agent with Self-Corrected Planning for Zero-Shot Vision-and-Language Navigation [73.78984332354636]
  CorNav is a novel zero-shot framework for vision-and-language navigation. It incorporates environmental feedback for refining future plans and adjusting its actions. It consistently outperforms all baselines in a zero-shot multi-task setting.
  arXiv  Detail & Related papers  (2023-06-17T11:44:04Z)
• NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models [17.495162643127003]
  We introduce the NavGPT to reveal the reasoning capability of GPT models in complex embodied scenes. NavGPT takes the textual descriptions of visual observations, navigation history, and future explorable directions as inputs to reason the agent's current status. We show that NavGPT is capable of generating high-quality navigational instructions from observations and actions along a path.
  arXiv  Detail & Related papers  (2023-05-26T14:41:06Z)
• Waypoint Models for Instruction-guided Navigation in Continuous Environments [68.2912740006109]
  We develop a class of language-conditioned waypoint prediction networks to examine this question. We measure task performance and estimated execution time on a profiled LoCoBot robot. Our models outperform prior work in VLN-CE and set a new state-of-the-art on the public leaderboard.
  arXiv  Detail & Related papers  (2021-10-05T17:55:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.