LocalNav: Distilling Frontier VLMs and Embodied RL for On-Device Object Goal Navigation

• URL: http://arxiv.org/abs/2606.27871v1
• Date: Fri, 26 Jun 2026 09:11:59 GMT
• Title: LocalNav: Distilling Frontier VLMs and Embodied RL for On-Device Object Goal Navigation
• Abstract summary: Vision Language Models (VLMs) have emerged in the robotic domain as a powerful tool that enables environmental perception with language context.<n>Yet, their computational footprint typically confines them to cloud execution, hindering low-latency inference with local deployment on resource-constrained robots.<n>We present a distillation strategy that transfers complex spatial-semantic reasoning from large frontier models into a lightweight, 4B- parameter local VLM for edge execution on embedded GPU devices.
• Score: 53.88286427894783
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision Language Models (VLMs) have emerged in the robotic domain as a powerful tool that enables environmental perception with language context, serving as a catalyst for open-vocabulary tasks like ObjectNav. Yet, their computational footprint typically confines them to cloud execution, hindering low-latency inference with local deployment on resource-constrained robots. To address this challenge, we present a distillation strategy that transfers complex spatial-semantic reasoning from large frontier models into a lightweight, 4B-parameter local VLM for edge execution on embedded GPU devices (e.g., Jetson Orin). We first establish a State of the Art (SotA), Scene Graph (SG)-based pipeline using Claude Sonnet 4.6, achieving a 39.7% Success Rate (SR) on the HM3D OVON benchmark. We then demonstrate that fine-tuning Qwen3.5-4B on just 500 frontier reasoning traces effectively enables navigation capabilities, yielding a SR of 34.5%, narrowing the gap to the performance of large cloud models. Finally, we introduce E-RLVR with Token Generation (TG) regularization to compress output sequence lengths for physical deployment while grounding the agent in its task. This downstream optimization reduces TG overhead by 72.1% and latency by 71.8%. Combined with quantization, this joint strategy yields a cumulative 82.8% reduction in overall inference latency without significantly sacrificing performance, presenting a viable paradigm for local, low-latency VLM execution on mobile robots.

Related papers

• On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability [57.03393882933515]
  We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and 51B parameters of n-gram embedding tables held off the accelerator.<n>On fourteen pre-training benchmarks the model leads the 397B-A17B predecessor on eight and trails it on the rest by at most 2.6 points, at 1/3 the activated parameters, 1/3 the training tokens, and roughly 1/9 the training FLOPs.
  arXiv  Detail & Related papers  (2026-08-31T06:35:07Z)
• Uni-LaViRA: Language-Vision-Robot Actions Translation for Unified Embodied Navigation [60.07205156194741]
  Embodied navigation requires an agent to map language and visual observations to a stream of spatial actions that drive a real robot through environments it has never seen.<n>We present Uni-LaViRA, a unified agentic architecture that extends the same insight to four task families and to four heterogeneous real robots.
  arXiv  Detail & Related papers  (2026-05-26T18:52:04Z)
• LightAVSeg: Lightweight Audio-Visual Segmentation [67.18006751024687]
  LightAVSeg replaces heavy attention with a decoupled design for semantic filtering and spatial grounding.<n>Experiments demonstrate it achieves a new state-of-the-art among lightweight methods.<n>It reaches 50.4 mIoU on the MS3 benchmark and enables efficient inference on a mobile processor.
  arXiv  Detail & Related papers  (2026-05-09T08:47:54Z)
• Escaping the Context Bottleneck: Active Context Curation for LLM Agents via Reinforcement Learning [6.577332846728808]
  We introduce a symbiotic framework that decouples context management from task execution.<n>Our architecture pairs a lightweight, specialized policy model, ContextCurator, with a powerful frozen foundation model, TaskExecutor.<n>On WebArena, our framework improves the success rate of Gemini-3.0-flash from 36.4% to 41.2% while reducing token consumption by 8.8%.<n>On DeepSearch, it achieves a 57.1% success rate, compared with 53.9%, while reducing token consumption by a factor of 8.
  arXiv  Detail & Related papers  (2026-04-13T13:39:17Z)
• TinyML Enhances CubeSat Mission Capabilities [8.462623779881566]
  The paper presents a Convolutional Neural Networks (ConvNets) model optimization and deployment pipeline for onboard image classification.<n>Our pipeline integrates structured iterative pruning, post-training INT8 quantization, and hardware-aware operator mapping to compress models.<n>We demonstrate an average reduction in RAM usage of 89.55% and Flash memory of 709% for the optimized models.
  arXiv  Detail & Related papers  (2026-03-20T17:51:07Z)
• GLANCE: Gaze-Led Attention Network for Compressed Edge-inference [10.229095428511654]
  Real-time object detection in AR/VR systems faces critical computational constraints, requiring sub-10,ms latency within tight power budgets.<n>Inspired by biological foveal vision, we propose a two-stage pipeline that combines differentiable weightless neural networks for ultra-efficient gaze estimation with attention-guided region-of-interest object detection.
  arXiv  Detail & Related papers  (2026-03-16T15:52:52Z)
• EdgeNav-QE: QLoRA Quantization and Dynamic Early Exit for LAM-based Navigation on Edge Devices [5.926544683744667]
  Large Action Models (LAMs) have shown immense potential in autonomous navigation by bridging high-level reasoning with low-level control.<n> deploying these multi-billion parameter models on edge devices remains a significant challenge due to memory constraints and latency requirements.<n>We propose EdgeNav-QE, a novel framework that integrates Quantized Low-Rank Adaptation (QLoRA) with a dynamic early-exit mechanism to optimize LAMs for real-time edge navigation.
  arXiv  Detail & Related papers  (2026-01-12T09:20:53Z)
• Open-Vocabulary Spatio-Temporal Scene Graph for Robot Perception and Teleoperation Planning [55.90805559207812]
  In dynamic remote scenes, transmission latency during bidirectional communication creates gaps between remote perceived states and operator intent.<n>We present a representation that enriches open-vocabulary perception with temporal dynamics and lightweight latency annotations.<n>We show that our method achieves 74 percent node accuracy on the Replica benchmark, outperforming Concept.Graph.
  arXiv  Detail & Related papers  (2025-09-27T04:31:24Z)
• A Novel Compression Framework for YOLOv8: Achieving Real-Time Aerial Object Detection on Edge Devices via Structured Pruning and Channel-Wise Distillation [0.0]
  We propose a novel three-stage compression pipeline for the YOLOv8 object detection model.<n>Sparsity-aware training, structured channel pruning, and Channel-Wise Knowledge Distillation (CWD) are used.<n>Experiments on the VisDrone dataset demonstrate the effectiveness of our approach across multiple YOLOv8 variants.
  arXiv  Detail & Related papers  (2025-09-16T10:11:59Z)
• Robustifying the Deployment of tinyML Models for Autonomous mini-vehicles [61.27933385742613]
  We propose a closed-loop learning flow for autonomous driving mini-vehicles that includes the target environment in-the-loop. We leverage a family of tinyCNNs to control the mini-vehicle, which learn in the target environment by imitating a computer vision algorithm, i.e., the expert. When running the family of CNNs, our solution outperforms any other implementation on the STM32L4 and k64f (Cortex-M4), reducing the latency by over 13x and the energy consummation by 92%.
  arXiv  Detail & Related papers  (2020-07-01T07:54:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.