LocalNav: Distilling Frontier VLMs and Embodied RL for On-Device Object Goal Navigation
Abstract Overview
This paper presents LocalNav, a scene-graph-based object-goal navigation system that aims to run vision-language reasoning directly on resource-constrained robots instead of relying on cloud-hosted frontier models. The method distills spatial-semantic reasoning from large VLMs into a lightweight 4B-parameter local model, using roughly 500 successful teacher reasoning traces collected in simulation. The system combines high-level VLM decisions over an explicit scene graph with a separate low-level navigation policy, and then applies embodied reinforcement learning with token-length regularization to reduce inference overhead. The paper evaluates the approach in simulation on HM3D OVON, profiles latency on a Jetson Orin AGX, and also demonstrates sequential real-world ObjectNav tasks in an apartment.
Novelty
The distinctive contribution is a deployment-first pipeline that transfers high-level ObjectNav reasoning from cloud frontier VLMs into a 4B local VLM while preserving a modular scene-graph navigation architecture. The work also introduces E-RLVR with token-generation regularization to explicitly optimize VLM outputs for on-device embodied execution, rather than only for task accuracy.
Results
In benchmark evaluation, the frontier-model version reaches 39.7% success rate on HM3D OVON, while the distilled Qwen3.5-4B model achieves 34.5%, leaving a relatively small gap to the cloud system. In the ablations, Claude-distilled Qwen reaches 47% SR before RL and 49% SR after SFT plus E-RLVR in the privileged-action setting, while output length is reduced by 72.1% and latency by 71.8%. With IQ4-XS quantization on Jetson Orin AGX, total average runtime per episode drops from 305.2 s for the SFT model to 52.5 s, for an overall 82.8% latency reduction without major performance loss.
Key Points
- LocalNav uses a modular scene-graph representation so the VLM makes sparse high-level decisions, while low-level control is handled by a separate navigation policy.
- A small amount of frontier-model supervision—about 500 successful reasoning traces—is enough to substantially improve a 4B local VLM for open-vocabulary ObjectNav.
- Embodied RL with brevity regularization, combined with quantization, makes the distilled model more practical for on-device deployment by sharply reducing token generation and end-to-end inference latency.