Aligning Knowledge Graph with Visual Perception for Object-goal Navigation

• URL: http://arxiv.org/abs/2402.18892v2
• Date: Fri, 26 Apr 2024 02:16:11 GMT
• Title: Aligning Knowledge Graph with Visual Perception for Object-goal Navigation
• Authors: Nuo Xu, Wen Wang, Rong Yang, Mengjie Qin, Zheyuan Lin, Wei Song, Chunlong Zhang, Jason Gu, Chao Li,
• Abstract summary: We propose the Aligning Knowledge Graph with Visual Perception (AKGVP) method for object-goal navigation. Our approach introduces continuous modeling of the hierarchical scene architecture and leverages visual-language pre-training to align natural language description with visual perception. The integration of a continuous knowledge graph architecture and multimodal feature alignment empowers the navigator with a remarkable zero-shot navigation capability.
• Score: 16.32780793344835
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Object-goal navigation is a challenging task that requires guiding an agent to specific objects based on first-person visual observations. The ability of agent to comprehend its surroundings plays a crucial role in achieving successful object finding. However, existing knowledge-graph-based navigators often rely on discrete categorical one-hot vectors and vote counting strategy to construct graph representation of the scenes, which results in misalignment with visual images. To provide more accurate and coherent scene descriptions and address this misalignment issue, we propose the Aligning Knowledge Graph with Visual Perception (AKGVP) method for object-goal navigation. Technically, our approach introduces continuous modeling of the hierarchical scene architecture and leverages visual-language pre-training to align natural language description with visual perception. The integration of a continuous knowledge graph architecture and multimodal feature alignment empowers the navigator with a remarkable zero-shot navigation capability. We extensively evaluate our method using the AI2-THOR simulator and conduct a series of experiments to demonstrate the effectiveness and efficiency of our navigator. Code available: https://github.com/nuoxu/AKGVP.

Related papers

• Augmented Commonsense Knowledge for Remote Object Grounding [67.30864498454805]
  We propose an augmented commonsense knowledge model (ACK) to leverage commonsense information as atemporal knowledge graph for improving agent navigation. ACK consists of knowledge graph-aware cross-modal and concept aggregation modules to enhance visual representation and visual-textual data alignment. We add a new pipeline for the commonsense-based decision-making process which leads to more accurate local action prediction.
  arXiv  Detail & Related papers  (2024-06-03T12:12:33Z)
• NavHint: Vision and Language Navigation Agent with a Hint Generator [31.322331792911598]
  We provide indirect supervision to the navigation agent through a hint generator that provides detailed visual descriptions. The hint generator assists the navigation agent in developing a global understanding of the visual environment. We evaluate our method on the R2R and R4R datasets and achieve state-of-the-art on several metrics.
  arXiv  Detail & Related papers  (2024-02-04T16:23:16Z)
• Zero-Shot Object Goal Visual Navigation With Class-Independent Relationship Network [3.0820097046465285]
  "Zero-shot" means that the target the agent needs to find is not trained during the training phase. We propose the Class-Independent Relationship Network (CIRN) to address the issue of coupling navigation ability with target features during training. Our method outperforms the current state-of-the-art approaches in the zero-shot object goal visual navigation task.
  arXiv  Detail & Related papers  (2023-10-15T16:42:14Z)
• Object Goal Navigation with Recursive Implicit Maps [92.6347010295396]
  We propose an implicit spatial map for object goal navigation. Our method significantly outperforms the state of the art on the challenging MP3D dataset. We deploy our model on a real robot and achieve encouraging object goal navigation results in real scenes.
  arXiv  Detail & Related papers  (2023-08-10T14:21:33Z)
• Learning Navigational Visual Representations with Semantic Map Supervision [85.91625020847358]
  We propose a navigational-specific visual representation learning method by contrasting the agent's egocentric views and semantic maps. Ego$2$-Map learning transfers the compact and rich information from a map, such as objects, structure and transition, to the agent's egocentric representations for navigation.
  arXiv  Detail & Related papers  (2023-07-23T14:01:05Z)
• How To Not Train Your Dragon: Training-free Embodied Object Goal Navigation with Semantic Frontiers [94.46825166907831]
  We present a training-free solution to tackle the object goal navigation problem in Embodied AI. Our method builds a structured scene representation based on the classic visual simultaneous localization and mapping (V-SLAM) framework. Our method propagates semantics on the scene graphs based on language priors and scene statistics to introduce semantic knowledge to the geometric frontiers.
  arXiv  Detail & Related papers  (2023-05-26T13:38:33Z)
• KERM: Knowledge Enhanced Reasoning for Vision-and-Language Navigation [61.08389704326803]
  Vision-and-language navigation (VLN) is the task to enable an embodied agent to navigate to a remote location following the natural language instruction in real scenes. Most of the previous approaches utilize the entire features or object-centric features to represent navigable candidates. We propose a Knowledge Enhanced Reasoning Model (KERM) to leverage knowledge to improve agent navigation ability.
  arXiv  Detail & Related papers  (2023-03-28T08:00:46Z)
• Visual Navigation with Spatial Attention [26.888916048408895]
  This work focuses on object goal visual navigation, aiming at finding the location of an object from a given class. We propose to learn the agent's policy using a reinforcement learning algorithm. Our key contribution is a novel attention probability model for visual navigation tasks.
  arXiv  Detail & Related papers  (2021-04-20T07:39:52Z)
• SOON: Scenario Oriented Object Navigation with Graph-based Exploration [102.74649829684617]
  The ability to navigate like a human towards a language-guided target from anywhere in a 3D embodied environment is one of the 'holy grail' goals of intelligent robots. Most visual navigation benchmarks focus on navigating toward a target from a fixed starting point, guided by an elaborate set of instructions that depicts step-by-step. This approach deviates from real-world problems in which human-only describes what the object and its surrounding look like and asks the robot to start navigation from anywhere.
  arXiv  Detail & Related papers  (2021-03-31T15:01:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.