GemNav: Discrete-Token Visual Robot Navigation using a Multimodal Large Language Model

• URL: http://arxiv.org/abs/2607.06882v1
• Date: Wed, 08 Jul 2026 00:46:57 GMT
• Title: GemNav: Discrete-Token Visual Robot Navigation using a Multimodal Large Language Model
• Abstract summary: We introduce GemNav, a visual robot navigation policy that adapts a frozen Multimodal Large Language Model.<n>On a single 8.7-hour open corpus, roughly three orders of magnitude smaller than competing training sets, the policy transfers zero-shot to four physically distinct unseen environments.<n> Conditioning on short image histories improves offline metrics but yields no robot benefit, pointing to a ceiling on what temporal context adds once pretrained vision features are in place.
• Score: 7.161190526180227
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Visual navigation policies built on large pretrained models have so far followed a common recipe: a dedicated visual encoder, a bespoke action head, and training on thousands of hours of cross-embodiment datasets. We ask whether this recipe is necessary. In this paper, we introduce GemNav, a visual robot navigation policy that adapts a frozen Multimodal Large Language Model (MLLM) for short-to-medium horizon waypoint navigation using Low-Rank Adaptation (LoRA) on the language tower alone, with no auxiliary visual encoder and no continuous regression head. Waypoints and categorical navigation signals share a single discrete token vocabulary generated by the language-model head, and a soft-decoded auxiliary loss recovers the metric structure that pure cross-entropy training discards. On a single 8.7-hour open corpus, roughly three orders of magnitude smaller than competing training sets, the policy transfers zero-shot to four physically distinct unseen environments and stops within 0.25-0.42m of the goal across 20 real-world trials covering an open carpark, an obstacle carpark, a long outdoor chemical yard, and an indoor warehouse. Conditioning on short image histories improves offline metrics but yields no robot benefit, pointing to a ceiling on what temporal context adds once pretrained vision features are in place. These results indicate that discrete-token adaptation of frozen MLLMs can provide a data-efficient, deployable alternative for foundation model robot navigation.

Related papers

• One MLLM, One Call: Efficient Zero-Shot Vision-and-Language Navigation via Spatial-Aware Waypoints [10.32396390031609]
  Vision-and-Language Navigation in Continuous Environments (VLN-CE) requires an embodied agent to navigate unseen environments by following natural language instructions.<n>We propose O2C-Nav, an efficient zero-shot navigation framework that calls only a single large model once per decision step.<n>Our approach introduces a training-free structured waypoint generator and a novel abstract representation that projects sparse, history-aware candidate waypoints directly onto RGB images as visual markers.
  arXiv  Detail & Related papers  (2026-09-06T08:49:10Z)
• LightNav-0: Eliciting VLM Spatial Intelligence for Generalist Embodied Navigation [52.91080323083409]
  Embodied navigation requires agents to translate heterogeneous goals and visual observations into actions across tasks, environments, and robot embodiments.<n>Here we present LightNav-0, a compact generalist embodied navigation model that elicits the spatial intelligence of a pretrained VLM and aligns it with navigation, without task-specific prediction heads.
  arXiv  Detail & Related papers  (2026-08-31T15:08:53Z)
• Offline Vision-Language Navigation with Geometric Goal Localization for Outdoor Environments [0.0]
  Foundation-model-based vision-language navigation (VLN) has advanced autonomous robot navigation by enabling robots to interpret natural-language instructions.<n>Existing VLN systems rely heavily on cloud-hosted foundation models for language understanding and semantic grounding.<n>This paper makes three contributions toward fully onboard VLN for outdoor environments.
  arXiv  Detail & Related papers  (2026-07-24T11:49:25Z)
• Qwen-RobotNav Technical Report: A Scalable Navigation Model Designed for an Agentic Navigation System [96.69286664036143]
  We present Qwen-RobotNav, a scalable navigation model built on Qwen-RobotNav.<n>We train Qwen-RobotNav on 15.6M samples; co-training with vision-language data prevents the collapse into reactive action-sequence mappers.<n>Qwen-RobotNav sets new state-of-the-art results across major navigation benchmarks.
  arXiv  Detail & Related papers  (2026-06-16T16:17:44Z)
• PlatonicNav: Unveiling Semantic Correspondence in Navigation with Platonic Topological Maps [14.35568539633215]
  Embodied visual navigation underpins a wide range of applications such as household service robotics, assistive robotics, and large-scale autonomous exploration.<n>We introduce PlatonicNav, a training-free framework whose Platonic Topological Map fuses geometric and semantic node distances from a self-supervised visual encoder, and grounds language goals via blind matching without any paired vision-language data.
  arXiv  Detail & Related papers  (2026-06-01T07:08:50Z)
• Uni-LaViRA: Language-Vision-Robot Actions Translation for Unified Embodied Navigation [60.07205156194741]
  Embodied navigation requires an agent to map language and visual observations to a stream of spatial actions that drive a real robot through environments it has never seen.<n>We present Uni-LaViRA, a unified agentic architecture that extends the same insight to four task families and to four heterogeneous real robots.
  arXiv  Detail & Related papers  (2026-05-26T18:52:04Z)
• ReasonNavi: Human-Inspired Global Map Reasoning for Zero-Shot Embodied Navigation [53.95797153529148]
  Embodied agents often struggle with efficient navigation because they rely primarily on partial egocentric observations.<n>We introduce ReasonNavi, a human-inspired framework that operationalizes this reason-then-act paradigm by coupling Multimodal Large Language Models (MLLMs) with deterministic planners.
  arXiv  Detail & Related papers  (2026-01-26T19:09:20Z)
• OpenNav: Open-World Navigation with Multimodal Large Language Models [8.41361699991122]
  Large language models (LLMs) have demonstrated strong common-sense reasoning abilities, making them promising for robotic navigation and planning tasks.<n>We aim to enable robots to interpret and decompose complex language instructions, ultimately synthesizing a sequence of trajectory points to complete diverse navigation tasks.<n>We validate our system on a Husky robot in both indoor and outdoor scenes, demonstrating its real-world robustness and applicability.
  arXiv  Detail & Related papers  (2025-07-24T02:05:28Z)
• FASTNav: Fine-tuned Adaptive Small-language-models Trained for Multi-point Robot Navigation [10.3997505825422]
  We propose FASTNav - a method for boosting lightweight language models (SLMs) for robot navigation. We train and evaluate models with FASTNav in both simulation and real robots, proving that we can deploy them with low cost, high accuracy and low response time.
  arXiv  Detail & Related papers  (2024-11-20T12:28:13Z)
• ViNT: A Foundation Model for Visual Navigation [52.2571739391896]
  Visual Navigation Transformer (ViNT) is a foundation model for vision-based robotic navigation. ViNT is trained with a general goal-reaching objective that can be used with any navigation dataset. It exhibits positive transfer, outperforming specialist models trained on singular datasets.
  arXiv  Detail & Related papers  (2023-06-26T16:57:03Z)
• LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action [76.71101507291473]
  We present a system, LM-Nav, for robotic navigation that enjoys the benefits of training on unannotated large datasets of trajectories. We show that such a system can be constructed entirely out of pre-trained models for navigation (ViNG), image-language association (CLIP), and language modeling (GPT-3), without requiring any fine-tuning or language-annotated robot data.
  arXiv  Detail & Related papers  (2022-07-10T10:41:50Z)
• Rapid Exploration for Open-World Navigation with Latent Goal Models [78.45339342966196]
  We describe a robotic learning system for autonomous exploration and navigation in diverse, open-world environments. At the core of our method is a learned latent variable model of distances and actions, along with a non-parametric topological memory of images. We use an information bottleneck to regularize the learned policy, giving us (i) a compact visual representation of goals, (ii) improved generalization capabilities, and (iii) a mechanism for sampling feasible goals for exploration.
  arXiv  Detail & Related papers  (2021-04-12T23:14:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.