Can Vision Foundation Models Navigate? Zero-Shot Real-World Evaluation and Lessons Learned

• URL: http://arxiv.org/abs/2603.25937v1
• Date: Thu, 26 Mar 2026 22:04:49 GMT
• Title: Can Vision Foundation Models Navigate? Zero-Shot Real-World Evaluation and Lessons Learned
• Abstract summary: Visual Navigation Models (VNMs) promise generalizable, robot navigation by learning from large-scale visual demonstrations.<n>We present a real-world evaluation of five state-of-the-art VNMs across two robot platforms and five environments spanning indoor and outdoor settings.
• Score: 5.561294055181353
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Visual Navigation Models (VNMs) promise generalizable, robot navigation by learning from large-scale visual demonstrations. Despite growing real-world deployment, existing evaluations rely almost exclusively on success rate, whether the robot reaches its goal, which conceals trajectory quality, collision behavior, and robustness to environmental change. We present a real-world evaluation of five state-of-the-art VNMs (GNM, ViNT, NoMaD, NaviBridger, and CrossFormer) across two robot platforms and five environments spanning indoor and outdoor settings. Beyond success rate, we combine path-based metrics with vision-based goal-recognition scores and assess robustness through controlled image perturbations (motion blur, sunflare). Our analysis uncovers three systematic limitations: (a) even architecturally sophisticated diffusion and transformer-based models exhibit frequent collisions, indicating limited geometric understanding; (b) models fail to discriminate between different locations that are perceptually similar, however some semantics differences are present, causing goal prediction errors in repetitive environments; and (c) performance degrades under distribution shift. We will publicly release our evaluation codebase and dataset to facilitate reproducible benchmarking of VNMs.

Related papers

• MemoBench: Benchmarking World Modeling in Dynamically Changing Environments [72.23517478870605]
  We introduce MemoBench, a diagnostic benchmark built around the disappear-and-reappear paradigm in dynamically changing environments.<n>We curate 360 ground-truth clips spanning synthetic and real-world scenes, and design an evaluation suite combining automated metrics with VQA-based assessment.
  arXiv  Detail & Related papers  (2026-06-25T20:37:39Z)
• VISTA: Scale-Aware Visual Navigation via Action History Conditioning [4.792216056979392]
  Vision Navigation Foundation Models (VNMs) promise end-to-end learned navigation policies capable of zero-shot deployment.<n>Current VNMs often struggle in visually repetitive environments that lack distinct features.<n>We present VISTA, which generalizes robustly to out-of-distribution environments, achieving 100% goal prediction accuracy in zero-shot, real-world deployment.
  arXiv  Detail & Related papers  (2026-06-15T21:01:30Z)
• WAM-Nav: Asymmetric Latent World-Action Modeling for Unified Visual Navigation [31.539730066120374]
  Visual navigation requires generating smooth and collision-free trajectories under complex geometric and physical constraints.<n>We propose WAM-Nav, a Latent World-Action Model for embodied visual navigation that jointly learns action generation and latent visual foresight.
  arXiv  Detail & Related papers  (2026-06-03T14:05:19Z)
• Multimodal embodiment-aware navigation transformer [5.673561896367136]
  ViLiNT is a multimodal, attention-based policy for goal navigation, trained on heterogeneous data from multiple platforms and environments.<n>We show that ViLiNT improves Success Rate on average by 166% over equivalent state-of-the-art vision-only baseline (NoMaD)<n>Results are confirmed through real-world deployments of a rover navigating in obstacle fields.
  arXiv  Detail & Related papers  (2026-04-21T09:32:59Z)
• ManipArena: Comprehensive Real-world Evaluation of Reasoning-Oriented Generalist Robot Manipulation [66.02142169323521]
  Vision-Language-Action models and world models have recently emerged as promising paradigms for general-purpose robotic intelligence.<n>Existing benchmarks are largely simulator-centric, which provide controllability but fail to capture the reality gap caused by perception noise.<n>We introduce ManipArena, a standardized evaluation framework designed to bridge simulation and real-world execution.
  arXiv  Detail & Related papers  (2026-03-30T15:06:41Z)
• 3DGSNav: Enhancing Vision-Language Model Reasoning for Object Navigation via Active 3D Gaussian Splatting [12.057873540714098]
  3DGSNav is a novel framework that embeds 3D Gaussian Splatting (3DGS) as persistent memory for vision-language models (VLMs) to enhance spatial reasoning.<n>3DGSNav incrementally constructs a 3DGS representation of the environment, enabling trajectory-guided free-viewpoint rendering of frontier-aware first-person views.<n>During navigation, a real-time object detector filters potential targets, while VLM-driven active viewpoint switching performs target re-verification.
  arXiv  Detail & Related papers  (2026-02-12T16:41:26Z)
• RADAR: Benchmarking Vision-Language-Action Generalization via Real-World Dynamics, Spatial-Physical Intelligence, and Autonomous Evaluation [76.22852262683746]
  We introduce RADAR, a benchmark designed to systematically evaluate VLA generalization under realistic conditions.<n>We apply RADAR to audit multiple state-of-the-art VLA models and uncover severe fragility beneath their apparent competence.
  arXiv  Detail & Related papers  (2026-02-11T16:08:30Z)
• Rethinking Visual-Language-Action Model Scaling: Alignment, Mixture, and Regularization [65.37179698521766]
  Vision-Language-Action (VLA) models show strong promise for generalist robot control.<n>It remains unclear whether -- and under what conditions -- the standard "scale data" recipe translates to robotics.<n>We present a systematic, controlled study of VLA scaling that revisits core training choices for pretraining across diverse robots.
  arXiv  Detail & Related papers  (2026-02-10T12:25:43Z)
• RoHOI: Robustness Benchmark for Human-Object Interaction Detection [84.78366452133514]
  Human-Object Interaction (HOI) detection is crucial for robot-human assistance, enabling context-aware support.<n>We introduce the first benchmark for HOI detection, evaluating model resilience under diverse challenges.<n>Our benchmark, RoHOI, includes 20 corruption types based on the HICO-DET and V-COCO datasets and a new robustness-focused metric.
  arXiv  Detail & Related papers  (2025-07-12T01:58:04Z)
• Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluation [54.3628937181904]
  Internal world models (WMs) enable agents to understand the world's state and predict transitions.<n>Recent large Vision-Language Models (VLMs), such as OpenAI o3, GPT-4o and Gemini, exhibit potential as general-purpose WMs.
  arXiv  Detail & Related papers  (2025-06-27T03:24:29Z)
• AdvReal: Physical Adversarial Patch Generation Framework for Security Evaluation of Object Detection Systems [13.653653250544004]
  We propose a unified joint adversarial training framework for both 2D and 3D domains.<n>We develop a realism enhancement mechanism that incorporates non-rigid deformation modeling and texture remapping.<n>Our method achieves an average attack success rate (ASR) of 70.13% on YOLOv12 in physical scenarios.
  arXiv  Detail & Related papers  (2025-05-22T08:54:03Z)
• Ground-level Viewpoint Vision-and-Language Navigation in Continuous Environments [10.953629652228024]
  Vision-and-Language Navigation (VLN) agents associate time-sequenced visual observations with corresponding instructions to make decisions.<n>In this paper, we address the mismatch between human-centric instructions and quadruped robots with a low-height field of view.<n>We propose a Ground-level Viewpoint Navigation (GVNav) approach to mitigate this issue.
  arXiv  Detail & Related papers  (2025-02-26T10:30:40Z)
• FovEx: Human-Inspired Explanations for Vision Transformers and Convolutional Neural Networks [8.659674736978555]
  We introduce Foveation-based Explanations (FovEx), a novel XAI method inspired by human vision.<n>Our method achieves state-of-the-art performance on both transformers and convolutional models.
  arXiv  Detail & Related papers  (2024-08-04T19:37:30Z)
• Counterfactual Vision-and-Language Navigation via Adversarial Path Sampling [65.99956848461915]
  Vision-and-Language Navigation (VLN) is a task where agents must decide how to move through a 3D environment to reach a goal.<n>One of the problems of the VLN task is data scarcity since it is difficult to collect enough navigation paths with human-annotated instructions for interactive environments.<n>We propose an adversarial-driven counterfactual reasoning model that can consider effective conditions instead of low-quality augmented data.
  arXiv  Detail & Related papers  (2019-11-17T18:02:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.