OpenFMNav: Towards Open-Set Zero-Shot Object Navigation via Vision-Language Foundation Models

• URL: http://arxiv.org/abs/2402.10670v2
• Date: Mon, 25 Mar 2024 02:52:43 GMT
• Title: OpenFMNav: Towards Open-Set Zero-Shot Object Navigation via Vision-Language Foundation Models
• Authors: Yuxuan Kuang, Hai Lin, Meng Jiang,
• Abstract summary: We propose OpenFMNav, an Open-set Foundation Model based framework for zero-shot object navigation. We first unleash the reasoning abilities of large language models to extract proposed objects from natural language instructions. We then leverage the generalizability of large vision language models to actively discover and detect candidate objects from the scene.
• Score: 16.50443396055173
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Object navigation (ObjectNav) requires an agent to navigate through unseen environments to find queried objects. Many previous methods attempted to solve this task by relying on supervised or reinforcement learning, where they are trained on limited household datasets with close-set objects. However, two key challenges are unsolved: understanding free-form natural language instructions that demand open-set objects, and generalizing to new environments in a zero-shot manner. Aiming to solve the two challenges, in this paper, we propose OpenFMNav, an Open-set Foundation Model based framework for zero-shot object Navigation. We first unleash the reasoning abilities of large language models (LLMs) to extract proposed objects from natural language instructions that meet the user's demand. We then leverage the generalizability of large vision language models (VLMs) to actively discover and detect candidate objects from the scene, building a Versatile Semantic Score Map (VSSM). Then, by conducting common sense reasoning on VSSM, our method can perform effective language-guided exploration and exploitation of the scene and finally reach the goal. By leveraging the reasoning and generalizing abilities of foundation models, our method can understand free-form human instructions and perform effective open-set zero-shot navigation in diverse environments. Extensive experiments on the HM3D ObjectNav benchmark show that our method surpasses all the strong baselines on all metrics, proving our method's effectiveness. Furthermore, we perform real robot demonstrations to validate our method's open-set-ness and generalizability to real-world environments.

Related papers

• Open-World Object Detection with Instance Representation Learning [1.8749305679160366]
  We propose a method to train an object detector that can both detect novel objects and extract semantically rich features in open-world conditions. Our method learns a robust and generalizable feature space, outperforming other OWOD-based feature extraction methods.
  arXiv  Detail & Related papers  (2024-09-24T13:13:34Z)
• One Map to Find Them All: Real-time Open-Vocabulary Mapping for Zero-shot Multi-Object Navigation [2.022249798290507]
  We introduce a new benchmark for zero-shot multi-object navigation. We build a reusable open-vocabulary feature map tailored for real-time object search. We demonstrate that it outperforms existing state-of-the-art approaches both on single and multi-object navigation tasks.
  arXiv  Detail & Related papers  (2024-09-18T07:44:08Z)
• Localizing Active Objects from Egocentric Vision with Symbolic World Knowledge [62.981429762309226]
  The ability to actively ground task instructions from an egocentric view is crucial for AI agents to accomplish tasks or assist humans virtually. We propose to improve phrase grounding models' ability on localizing the active objects by: learning the role of objects undergoing change and extracting them accurately from the instructions. We evaluate our framework on Ego4D and Epic-Kitchens datasets.
  arXiv  Detail & Related papers  (2023-10-23T16:14:05Z)
• Object Goal Navigation with Recursive Implicit Maps [92.6347010295396]
  We propose an implicit spatial map for object goal navigation. Our method significantly outperforms the state of the art on the challenging MP3D dataset. We deploy our model on a real robot and achieve encouraging object goal navigation results in real scenes.
  arXiv  Detail & Related papers  (2023-08-10T14:21:33Z)
• Can an Embodied Agent Find Your "Cat-shaped Mug"? LLM-Guided Exploration for Zero-Shot Object Navigation [58.3480730643517]
  We present LGX, a novel algorithm for Language-Driven Zero-Shot Object Goal Navigation (L-ZSON) Our approach makes use of Large Language Models (LLMs) for this task. We achieve state-of-the-art zero-shot object navigation results on RoboTHOR with a success rate (SR) improvement of over 27% over the current baseline.
  arXiv  Detail & Related papers  (2023-03-06T20:19:19Z)
• ESC: Exploration with Soft Commonsense Constraints for Zero-shot Object Navigation [75.13546386761153]
  We present a novel zero-shot object navigation method, Exploration with Soft Commonsense constraints (ESC) ESC transfers commonsense knowledge in pre-trained models to open-world object navigation without any navigation experience. Experiments on MP3D, HM3D, and RoboTHOR benchmarks show that our ESC method improves significantly over baselines.
  arXiv  Detail & Related papers  (2023-01-30T18:37:32Z)
• PONI: Potential Functions for ObjectGoal Navigation with Interaction-free Learning [125.22462763376993]
  We propose Potential functions for ObjectGoal Navigation with Interaction-free learning (PONI) PONI disentangles the skills of where to look?' for an object and how to navigate to (x, y)?'
  arXiv  Detail & Related papers  (2022-01-25T01:07:32Z)
• MOCA: A Modular Object-Centric Approach for Interactive Instruction Following [19.57344182656879]
  We propose a modular architecture that decouples the task into visual perception and action policy. We evaluate our method on the ALFRED benchmark and empirically validate that it outperforms prior arts.
  arXiv  Detail & Related papers  (2020-12-06T07:59:22Z)
• Object Goal Navigation using Goal-Oriented Semantic Exploration [98.14078233526476]
  This work studies the problem of object goal navigation which involves navigating to an instance of the given object category in unseen environments. We propose a modular system called, Goal-Oriented Semantic Exploration' which builds an episodic semantic map and uses it to explore the environment efficiently.
  arXiv  Detail & Related papers  (2020-07-01T17:52:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.