NaVILA: Legged Robot Vision-Language-Action Model for Navigation

• URL: http://arxiv.org/abs/2412.04453v2
• Date: Mon, 17 Feb 2025 18:27:27 GMT
• Title: NaVILA: Legged Robot Vision-Language-Action Model for Navigation
• Abstract summary: It is non-trivial to translate human language instructions all the way to low-level leg joint actions.<n>We propose NaVILA, a 2-level framework that unifies a Vision-Language-Action model (VLA) with locomotion skills.<n>NaVILA substantially improves previous approaches on existing benchmarks.
• Score: 60.00462044102051
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper proposes to solve the problem of Vision-and-Language Navigation with legged robots, which not only provides a flexible way for humans to command but also allows the robot to navigate through more challenging and cluttered scenes. However, it is non-trivial to translate human language instructions all the way to low-level leg joint actions. We propose NaVILA, a 2-level framework that unifies a Vision-Language-Action model (VLA) with locomotion skills. Instead of directly predicting low-level actions from VLA, NaVILA first generates mid-level actions with spatial information in the form of language, (e.g., "moving forward 75cm"), which serves as an input for a visual locomotion RL policy for execution. NaVILA substantially improves previous approaches on existing benchmarks. The same advantages are demonstrated in our newly developed benchmarks with IsaacLab, featuring more realistic scenes, low-level controls, and real-world robot experiments. We show more results at https://navila-bot.github.io/

Related papers

• StellaVLA: In-Context Structured Demonstration for Generalizable Vision-Language-Action Models [37.45050514108603]
  We present StellaVLA, a framework that adapts at test time by conditioning on a single retrieved demonstration.<n>An automated offline pipeline converts each raw trajectory into a structured demonstration, e.g., a task plan, sub-goal descriptions, and verbalized 3D motion, at zero human-annotation cost.<n>A parallel dual-training design internalizes this reasoning during training through a joint action-and-language objective, while inference uses the action expert alone.
  arXiv  Detail & Related papers  (2026-08-12T05:30:53Z)
• Generalizable VLA Finetuning via Representation Anchoring and Language-Action Alignment [42.503300854478795]
  Anchor-Align improves vision-language-action policies on robot demonstrations.<n>It distills layer-wise representations from a frozen VLM copy to prevent this drift.<n>It converts each action target into a discrete motion-direction label and jointly trains language and action prediction.
  arXiv  Detail & Related papers  (2026-07-15T04:13:54Z)
• Uni-LaViRA: Language-Vision-Robot Actions Translation for Unified Embodied Navigation [60.07205156194741]
  Embodied navigation requires an agent to map language and visual observations to a stream of spatial actions that drive a real robot through environments it has never seen.<n>We present Uni-LaViRA, a unified agentic architecture that extends the same insight to four task families and to four heterogeneous real robots.
  arXiv  Detail & Related papers  (2026-05-26T18:52:04Z)
• AwareVLN: Reasoning with Self-awareness for Vision-Language Navigation [78.80536515102305]
  Vision-and-Language Navigation (VLN) requires an agent to ground language instructions to its own movement within a visual environment.<n>We propose AwareVLN, a novel framework that equips the navigation model with a self-aware reasoning mechanism.<n>Our approach features two key innovations: (1) a structural reasoning module that fosters spatial and task-oriented self-awareness, and (2) an automatic data engine with progress division for effective training.
  arXiv  Detail & Related papers  (2026-05-21T17:58:26Z)
• LACY: A Vision-Language Model-based Language-Action Cycle for Self-Improving Robotic Manipulation [11.419077130835829]
  We introduce LACY (Language-Action Cycle), a unified framework that learns bidirectional mappings within a single vision-language model.<n>We show that LACY improves task success rates by 56.46% on average and yields more robust language-action grounding for robotic manipulation.
  arXiv  Detail & Related papers  (2025-11-04T04:02:51Z)
• LLaRA: Supercharging Robot Learning Data for Vision-Language Policy [56.505551117094534]
  We introduce LLaRA: Large Language and Robotics Assistant, a framework that formulates robot action policy as visuo-textual conversations. First, we present an automated pipeline to generate conversation-style instruction tuning data for robots from existing behavior cloning datasets. We show that a VLM finetuned with a limited amount of such datasets can produce meaningful action decisions for robotic control.
  arXiv  Detail & Related papers  (2024-06-28T17:59:12Z)
• RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics [46.63773228934993]
  We introduce an automatic synthetic data generation pipeline that instruction-tunes vision language models (VLMs) to robotic domains and needs. Using the pipeline, we train RoboPoint, a VLM that predicts image keypoint affordances given language instructions. Our experiments demonstrate that RoboPoint outperforms state-of-the-art VLMs by 21.8% in the accuracy of predicting spatial affordance and by 30.5% in the success rate of downstream tasks.
  arXiv  Detail & Related papers  (2024-06-15T19:22:51Z)
• Yell At Your Robot: Improving On-the-Fly from Language Corrections [84.09578841663195]
  We show that high-level policies can be readily supervised with human feedback in the form of language corrections. This framework enables robots not only to rapidly adapt to real-time language feedback, but also incorporate this feedback into an iterative training scheme.
  arXiv  Detail & Related papers  (2024-03-19T17:08:24Z)
• NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigation [23.72290930234063]
  NaVid is a video-based large vision language model (VLM) for vision-and-language navigation. NaVid achieves state-of-the-art performance in simulation environments and the real world, demonstrating superior cross-dataset and Sim2Real transfer.
  arXiv  Detail & Related papers  (2024-02-24T16:39:16Z)
• LangNav: Language as a Perceptual Representation for Navigation [63.90602960822604]
  We explore the use of language as a perceptual representation for vision-and-language navigation (VLN) Our approach uses off-the-shelf vision systems for image captioning and object detection to convert an agent's egocentric panoramic view at each time step into natural language descriptions.
  arXiv  Detail & Related papers  (2023-10-11T20:52:30Z)
• RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control [140.48218261864153]
  We study how vision-language models trained on Internet-scale data can be incorporated directly into end-to-end robotic control. Our approach leads to performant robotic policies and enables RT-2 to obtain a range of emergent capabilities from Internet-scale training.
  arXiv  Detail & Related papers  (2023-07-28T21:18:02Z)
• ViNL: Visual Navigation and Locomotion Over Obstacles [36.46953494419389]
  We present Visual Navigation and Locomotion over obstacles (ViNL) It enables a quadrupedal robot to navigate unseen apartments while stepping over small obstacles that lie in its path. ViNL consists of: (1) a visual navigation policy that outputs linear and angular velocity commands that guides the robot to a goal coordinate in unfamiliar indoor environments; and (2) a visual locomotion policy that controls the robot's joints to avoid stepping on obstacles while following provided velocity commands.
  arXiv  Detail & Related papers  (2022-10-26T15:38:28Z)
• LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action [76.71101507291473]
  We present a system, LM-Nav, for robotic navigation that enjoys the benefits of training on unannotated large datasets of trajectories. We show that such a system can be constructed entirely out of pre-trained models for navigation (ViNG), image-language association (CLIP), and language modeling (GPT-3), without requiring any fine-tuning or language-annotated robot data.
  arXiv  Detail & Related papers  (2022-07-10T10:41:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.