Prioritized Semantic Learning for Zero-shot Instance Navigation

• URL: http://arxiv.org/abs/2403.11650v2
• Date: Tue, 16 Jul 2024 18:13:07 GMT
• Title: Prioritized Semantic Learning for Zero-shot Instance Navigation
• Authors: Xinyu Sun, Lizhao Liu, Hongyan Zhi, Ronghe Qiu, Junwei Liang,
• Abstract summary: We study zero-shot instance navigation, in which the agent navigates to a specific object without using object annotations for training. We propose a Prioritized Semantic Learning (PSL) method to improve the semantic understanding ability of navigation agents. Our PSL agent outperforms the previous state-of-the-art by 66% on zero-shot ObjectNav in terms of success rate and is also superior on the new InstanceNav task.
• Score: 2.537056548731396
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We study zero-shot instance navigation, in which the agent navigates to a specific object without using object annotations for training. Previous object navigation approaches apply the image-goal navigation (ImageNav) task (go to the location of an image) for pretraining, and transfer the agent to achieve object goals using a vision-language model. However, these approaches lead to issues of semantic neglect, where the model fails to learn meaningful semantic alignments. In this paper, we propose a Prioritized Semantic Learning (PSL) method to improve the semantic understanding ability of navigation agents. Specifically, a semantic-enhanced PSL agent is proposed and a prioritized semantic training strategy is introduced to select goal images that exhibit clear semantic supervision and relax the reward function from strict exact view matching. At inference time, a semantic expansion inference scheme is designed to preserve the same granularity level of the goal semantic as training. Furthermore, for the popular HM3D environment, we present an Instance Navigation (InstanceNav) task that requires going to a specific object instance with detailed descriptions, as opposed to the Object Navigation (ObjectNav) task where the goal is defined merely by the object category. Our PSL agent outperforms the previous state-of-the-art by 66% on zero-shot ObjectNav in terms of success rate and is also superior on the new InstanceNav task. Code will be released at https://github.com/XinyuSun/PSL-InstanceNav.

Related papers

• SG-Nav: Online 3D Scene Graph Prompting for LLM-based Zero-shot Object Navigation [83.4599149936183]
  Existing zero-shot object navigation methods prompt LLM with the text of spatially closed objects. We propose to represent the observed scene with 3D scene graph. We conduct extensive experiments on MP3D, HM3D and RoboTHOR environments, where SG-Nav surpasses previous state-of-the-art zero-shot methods by more than 10% SR on all benchmarks.
  arXiv  Detail & Related papers  (2024-10-10T17:57:19Z)
• Instance-aware Exploration-Verification-Exploitation for Instance ImageGoal Navigation [88.84058353659107]
  Instance ImageGoal Navigation (IIN) aims to navigate to a specified object depicted by a goal image in an unexplored environment. We propose a new modular navigation framework named Instance-aware Exploration-Verification-Exploitation (IEVE) for instance-level image goal navigation. Our method surpasses previous state-of-the-art work, with a classical segmentation model (0.684 vs. 0.561 success) or a robust model (0.702 vs. 0.561 success)
  arXiv  Detail & Related papers  (2024-02-25T07:59:10Z)
• Zero-Shot Object Goal Visual Navigation With Class-Independent Relationship Network [3.0820097046465285]
  "Zero-shot" means that the target the agent needs to find is not trained during the training phase. We propose the Class-Independent Relationship Network (CIRN) to address the issue of coupling navigation ability with target features during training. Our method outperforms the current state-of-the-art approaches in the zero-shot object goal visual navigation task.
  arXiv  Detail & Related papers  (2023-10-15T16:42:14Z)
• Object Goal Navigation with Recursive Implicit Maps [92.6347010295396]
  We propose an implicit spatial map for object goal navigation. Our method significantly outperforms the state of the art on the challenging MP3D dataset. We deploy our model on a real robot and achieve encouraging object goal navigation results in real scenes.
  arXiv  Detail & Related papers  (2023-08-10T14:21:33Z)
• How To Not Train Your Dragon: Training-free Embodied Object Goal Navigation with Semantic Frontiers [94.46825166907831]
  We present a training-free solution to tackle the object goal navigation problem in Embodied AI. Our method builds a structured scene representation based on the classic visual simultaneous localization and mapping (V-SLAM) framework. Our method propagates semantics on the scene graphs based on language priors and scene statistics to introduce semantic knowledge to the geometric frontiers.
  arXiv  Detail & Related papers  (2023-05-26T13:38:33Z)
• Instance-Specific Image Goal Navigation: Training Embodied Agents to Find Object Instances [90.61897965658183]
  We consider the problem of embodied visual navigation given an image-goal (ImageNav) Unlike related navigation tasks, ImageNav does not have a standardized task definition which makes comparison across methods difficult. We present the Instance-specific ImageNav task (ImageNav) to address these limitations.
  arXiv  Detail & Related papers  (2022-11-29T02:29:35Z)
• Learning to Map for Active Semantic Goal Navigation [40.193928212509356]
  We propose a novel framework that actively learns to generate semantic maps outside the field of view of the agent. We show how different objectives can be defined by balancing exploration with exploitation. Our method is validated in the visually realistic environments offered by the Matterport3D dataset.
  arXiv  Detail & Related papers  (2021-06-29T18:01:30Z)
• SSCNav: Confidence-Aware Semantic Scene Completion for Visual Semantic Navigation [22.0915442335966]
  This paper focuses on visual semantic navigation, the task of producing actions for an active agent to navigate to a specified target object category in an unknown environment. We introduce SSCNav, an algorithm that explicitly models scene priors using a confidence-aware semantic scene completion module. Our experiments demonstrate that the proposed scene completion module improves the efficiency of the downstream navigation policies.
  arXiv  Detail & Related papers  (2020-12-08T15:59:47Z)
• ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objects [119.46959413000594]
  This document summarizes the consensus recommendations of a working group on ObjectNav. We make recommendations on subtle but important details of evaluation criteria. We provide a detailed description of the instantiation of these recommendations in challenges organized at the Embodied AI workshop at CVPR 2020.
  arXiv  Detail & Related papers  (2020-06-23T17:18:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.