Slow Brain, Fast Planner: Latency-Resilient VLM-Augmented Urban Navigation

• URL: http://arxiv.org/abs/2606.20458v1
• Date: Thu, 18 Jun 2026 16:40:07 GMT
• Title: Slow Brain, Fast Planner: Latency-Resilient VLM-Augmented Urban Navigation
• Abstract summary: Learning-based planners for sidewalk navigation can generate diverse candidate trajectories in real time.<n>But their scoring functions often fail to select the best trajectory in challenging situations.<n>We propose a VLM-Planner interface that uses a VLM to select a candidate index from the planner's proposal set and then fuse it with the planner's initial output.
• Score: 37.49678227133038
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Learning-based planners for sidewalk navigation can generate diverse candidate trajectories in real time, yet their scoring functions often fail to select the best trajectory in challenging situations, outputting trajectories that make the mobile robot drive onto grass, toward pedestrians, or in the wrong direction, even when better candidates exist in the same set. We call this the trajectory scoring gap: in real-world sidewalk navigation, the gap between an anchor-based planner's top choice and the best possible candidate is substantial, likely due to limited high-level scene understanding capability of the planner. Rather than replacing the planner with an end-to-end Vision-Language-Action model, we propose a VLM-Planner interface that uses a VLM to select a candidate index from the planner's proposal set and then fuse it with the planner's initial output. However, VLMs take 1--3s per query and so cannot directly drive a 5--20Hz control loop. We contribute a training-free, latency-resilient trajectory-level fusion layer that turns a stale VLM selection into real-time planner scoring via geometric similarity with exponential decay. On $\sim$2,000 challenging real-world scenarios (e.g., junctions, pedestrian encounters), VLM selection achieves 30% ADE reduction versus the planner's best selection, while the planner remains competitive in routine situations. In simulation, Score Fusion maintains >80% success rate with delays up to 5s. We demonstrate the full system on a mobile robot navigating challenging campus sidewalks with varied network latency.

Related papers

• LWDrive: Layer-Wise World-Model-Guided Vision-Language Model Planning for Autonomous Driving [13.964957969409655]
  We develop the Layer-Wise World-Model-Guided Driving framework (LWDrive)<n>LWDrive is a VLM planning framework that refines coarse trajectories through layer-wise world-model guidance.<n> Experiments show that LWDrive achieves a score of 92.0 on the NAVSIM benchmark and 89.6 on NAVSIM-v2.
  arXiv  Detail & Related papers  (2026-06-29T07:17:13Z)
• Test-Time Trajectory Optimization for Autonomous Driving [77.87162309994567]
  End-to-end planners for autonomous driving typically generate a set of candidate trajectories, score each one, and return the highest-scoring candidate.<n>We treat the scorer as a learned trajectory-level reward function and search for trajectories that maximize it.<n>Our method, TOAD, runs the Cross-Entropy Method at test time, warm-started from the planner's proposals.
  arXiv  Detail & Related papers  (2026-06-05T11:39:27Z)
• LLaViDA: A Large Language Vision Driving Assistant for Explicit Reasoning and Enhanced Trajectory Planning [28.59507336524504]
  Trajectory planning is a fundamental yet challenging component of autonomous driving.<n>We propose LLaViDA, a Large Language Vision Driving Assistant that leverages a Vision-Language Model (VLM) for object motion prediction.<n>On the NuScenes benchmark, LLaViDA surpasses state-of-the-art end-to-end and other recent VLM/LLM-based baselines in open-loop trajectory planning task.
  arXiv  Detail & Related papers  (2025-12-20T04:38:35Z)
• Distilling Multi-modal Large Language Models for Autonomous Driving [64.63127269187814]
  Recent end-to-end autonomous driving systems leverage large language models (LLMs) as planners to improve generalizability to rare events.<n>We propose DiMA, an end-to-end autonomous driving system that maintains the efficiency of an LLM-free (or vision-based) planner while leveraging the world knowledge of an LLM.<n>Training with DiMA results in a 37% reduction in the L2 trajectory error and an 80% reduction in the collision rate of the vision-based planner, as well as a 44% trajectory error reduction in longtail scenarios.
  arXiv  Detail & Related papers  (2025-01-16T18:59:53Z)
• SCoTT: Strategic Chain-of-Thought Tasking for Wireless-Aware Robot Navigation in Digital Twins [78.53885607559958]
  We propose SCoTT, a wireless-aware path planning framework.<n>We show that SCoTT achieves path gains within 2% of DP-WA* while consistently generating shorter trajectories.<n>We also show the practical viability of our approach by deploying SCoTT as a ROS node within Gazebo simulations.
  arXiv  Detail & Related papers  (2024-11-27T10:45:49Z)
• Affordances-Oriented Planning using Foundation Models for Continuous Vision-Language Navigation [64.84996994779443]
  We propose a novel Affordances-Oriented Planner for continuous vision-language navigation (VLN) task. Our AO-Planner integrates various foundation models to achieve affordances-oriented low-level motion planning and high-level decision-making. Experiments on the challenging R2R-CE and RxR-CE datasets show that AO-Planner achieves state-of-the-art zero-shot performance.
  arXiv  Detail & Related papers  (2024-07-08T12:52:46Z)
• 1st Place Solutions for RxR-Habitat Vision-and-Language Navigation Competition (CVPR 2022) [28.5740809300599]
  We present a modular plan-and-control approach for the problem of Vision-and-Language Navigation in Continuous Environments (VLN-CE) Our model consists of three modules: the candidate waypoints predictor (CWP), the history enhanced planner and the tryout controller. Our model won the RxR-Habitat Competition 2022, with 48% and 90% relative improvements over existing methods on NDTW and SR metrics respectively.
  arXiv  Detail & Related papers  (2022-06-23T10:36:53Z)
• Learning Forward Dynamics Model and Informed Trajectory Sampler for Safe Quadruped Navigation [1.2783783498844021]
  A typical SOTA system is composed of four main modules -- mapper, global planner, local planner, and command-tracking controller. We build a robust and safe local planner which is designed to generate a velocity plan to track a coarsely planned path from the global planner. Using our framework, a quadruped robot can autonomously navigate in various complex environments without a collision and generate a smoother command plan compared to the baseline method.
  arXiv  Detail & Related papers  (2022-04-19T04:01:44Z)
• End-to-end Interpretable Neural Motion Planner [78.69295676456085]
  We propose a neural motion planner (NMP) for learning to drive autonomously in complex urban scenarios. We design a holistic model that takes as input raw LIDAR data and a HD map and produces interpretable intermediate representations. We demonstrate the effectiveness of our approach in real-world driving data captured in several cities in North America.
  arXiv  Detail & Related papers  (2021-01-17T14:16:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.