Uni-NaVid: A Video-based Vision-Language-Action Model for Unifying Embodied Navigation Tasks

• URL: http://arxiv.org/abs/2412.06224v2
• Date: Thu, 06 Feb 2025 10:14:36 GMT
• Title: Uni-NaVid: A Video-based Vision-Language-Action Model for Unifying Embodied Navigation Tasks
• Authors: Jiazhao Zhang, Kunyu Wang, Shaoan Wang, Minghan Li, Haoran Liu, Songlin Wei, Zhongyuan Wang, Zhizheng Zhang, He Wang,
• Abstract summary: Existing models for embodied navigation fall short of serving as practical generalists in the real world. We present Uni-NaVid, the first video-based vision-language-action model designed to unify diverse embodied navigation tasks. Uni-NaVid achieves this by the input and output data configurations for all commonly used embodied navigation tasks.
• Score: 24.690910258151693
• License:
• Abstract: A practical navigation agent must be capable of handling a wide range of interaction demands, such as following instructions, searching objects, answering questions, tracking people, and more. Existing models for embodied navigation fall short of serving as practical generalists in the real world, as they are often constrained by specific task configurations or pre-defined maps with discretized waypoints. In this work, we present Uni-NaVid, the first video-based vision-language-action (VLA) model designed to unify diverse embodied navigation tasks and enable seamless navigation for mixed long-horizon tasks in unseen real-world environments. Uni-NaVid achieves this by harmonizing the input and output data configurations for all commonly used embodied navigation tasks and thereby integrating all tasks in one model. For training Uni-NaVid, we collect 3.6 million navigation data samples in total from four essential navigation sub-tasks and foster synergy in learning across them. Extensive experiments on comprehensive navigation benchmarks clearly demonstrate the advantages of unification modeling in Uni-NaVid and show it achieves state-of-the-art performance. Additionally, real-world experiments confirm the model's effectiveness and efficiency, shedding light on its strong generalizability.

Related papers

• Leveraging Large Language Model-based Room-Object Relationships Knowledge for Enhancing Multimodal-Input Object Goal Navigation [11.510823733292519]
  We propose a data-driven, modular-based approach, trained on a dataset that incorporates common-sense knowledge of object-to-room relationships extracted from a large language model. The results in the Habitat simulator demonstrate that our framework outperforms the baseline by an average of 10.6% in the efficiency metric, Success weighted by Path Length (SPL).
  arXiv  Detail & Related papers  (2024-03-21T06:32:36Z)
• Towards Learning a Generalist Model for Embodied Navigation [24.816490551945435]
  We propose the first generalist model for embodied navigation, NaviLLM. It adapts LLMs to embodied navigation by introducing schema-based instruction. We conduct extensive experiments to evaluate the performance and generalizability of our model.
  arXiv  Detail & Related papers  (2023-12-04T16:32:51Z)
• $A^2$Nav: Action-Aware Zero-Shot Robot Navigation by Exploiting Vision-and-Language Ability of Foundation Models [89.64729024399634]
  We study the task of zero-shot vision-and-language navigation (ZS-VLN), a practical yet challenging problem in which an agent learns to navigate following a path described by language instructions. Normally, the instructions have complex grammatical structures and often contain various action descriptions. How to correctly understand and execute these action demands is a critical problem, and the absence of annotated data makes it even more challenging.
  arXiv  Detail & Related papers  (2023-08-15T19:01:19Z)
• Learning Navigational Visual Representations with Semantic Map Supervision [85.91625020847358]
  We propose a navigational-specific visual representation learning method by contrasting the agent's egocentric views and semantic maps. Ego$2$-Map learning transfers the compact and rich information from a map, such as objects, structure and transition, to the agent's egocentric representations for navigation.
  arXiv  Detail & Related papers  (2023-07-23T14:01:05Z)
• ViNT: A Foundation Model for Visual Navigation [52.2571739391896]
  Visual Navigation Transformer (ViNT) is a foundation model for vision-based robotic navigation. ViNT is trained with a general goal-reaching objective that can be used with any navigation dataset. It exhibits positive transfer, outperforming specialist models trained on singular datasets.
  arXiv  Detail & Related papers  (2023-06-26T16:57:03Z)
• ESC: Exploration with Soft Commonsense Constraints for Zero-shot Object Navigation [75.13546386761153]
  We present a novel zero-shot object navigation method, Exploration with Soft Commonsense constraints (ESC) ESC transfers commonsense knowledge in pre-trained models to open-world object navigation without any navigation experience. Experiments on MP3D, HM3D, and RoboTHOR benchmarks show that our ESC method improves significantly over baselines.
  arXiv  Detail & Related papers  (2023-01-30T18:37:32Z)
• Towards Versatile Embodied Navigation [120.73460380993305]
  Vienna is a versatile embodied navigation agent that simultaneously learns to perform the four navigation tasks with one model. We empirically demonstrate that, compared with learning each visual navigation task individually, our agent achieves comparable or even better performance with reduced complexity.
  arXiv  Detail & Related papers  (2022-10-30T11:53:49Z)
• Zero Experience Required: Plug & Play Modular Transfer Learning for Semantic Visual Navigation [97.17517060585875]
  We present a unified approach to visual navigation using a novel modular transfer learning model. Our model can effectively leverage its experience from one source task and apply it to multiple target tasks. Our approach learns faster, generalizes better, and outperforms SoTA models by a significant margin.
  arXiv  Detail & Related papers  (2022-02-05T00:07:21Z)
• MultiON: Benchmarking Semantic Map Memory using Multi-Object Navigation [23.877609358505268]
  Recent work shows that map-like memory is useful for long-horizon navigation tasks. We propose the multiON task, which requires navigation to an episode-specific sequence of objects in a realistic environment. We examine how a variety of agent models perform across a spectrum of navigation task complexities.
  arXiv  Detail & Related papers  (2020-12-07T18:42:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.