End-to-End (Instance)-Image Goal Navigation through Correspondence as an Emergent Phenomenon

• URL: http://arxiv.org/abs/2309.16634v1
• Date: Thu, 28 Sep 2023 17:41:17 GMT
• Title: End-to-End (Instance)-Image Goal Navigation through Correspondence as an Emergent Phenomenon
• Authors: Guillaume Bono, Leonid Antsfeld, Boris Chidlovskii, Philippe Weinzaepfel, Christian Wolf
• Abstract summary: We propose a new dual encoder with a large-capacity binocular ViT model and show that correspondence solutions naturally emerge from the training signals. Experiments show significant improvements and SOTA performance on the two benchmarks, ImageNav and the Instance-ImageNav variant.
• Score: 27.252343068970852
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Most recent work in goal oriented visual navigation resorts to large-scale machine learning in simulated environments. The main challenge lies in learning compact representations generalizable to unseen environments and in learning high-capacity perception modules capable of reasoning on high-dimensional input. The latter is particularly difficult when the goal is not given as a category ("ObjectNav") but as an exemplar image ("ImageNav"), as the perception module needs to learn a comparison strategy requiring to solve an underlying visual correspondence problem. This has been shown to be difficult from reward alone or with standard auxiliary tasks. We address this problem through a sequence of two pretext tasks, which serve as a prior for what we argue is one of the main bottleneck in perception, extremely wide-baseline relative pose estimation and visibility prediction in complex scenes. The first pretext task, cross-view completion is a proxy for the underlying visual correspondence problem, while the second task addresses goal detection and finding directly. We propose a new dual encoder with a large-capacity binocular ViT model and show that correspondence solutions naturally emerge from the training signals. Experiments show significant improvements and SOTA performance on the two benchmarks, ImageNav and the Instance-ImageNav variant, where camera intrinsics and height differ between observation and goal.

Related papers

• DeTra: A Unified Model for Object Detection and Trajectory Forecasting [68.85128937305697]
  Our approach formulates the union of the two tasks as a trajectory refinement problem. To tackle this unified task, we design a refinement transformer that infers the presence, pose, and multi-modal future behaviors of objects. In our experiments, we observe that ourmodel outperforms the state-of-the-art on Argoverse 2 Sensor and Open dataset.
  arXiv  Detail & Related papers  (2024-06-06T18:12:04Z)
• A Simple yet Effective Network based on Vision Transformer for Camouflaged Object and Salient Object Detection [33.30644598646274]
  We propose a simple yet effective network (SENet) based on vision Transformer (ViT) To enhance the Transformer's ability to model local information, we propose a local information capture module (LICM) We also propose a dynamic weighted loss (DW loss) based on Binary Cross-Entropy (BCE) and Intersection over Union (IoU) loss, which guides the network to pay more attention to those smaller and more difficult-to-find target objects.
  arXiv  Detail & Related papers  (2024-02-29T07:29:28Z)
• How To Not Train Your Dragon: Training-free Embodied Object Goal Navigation with Semantic Frontiers [94.46825166907831]
  We present a training-free solution to tackle the object goal navigation problem in Embodied AI. Our method builds a structured scene representation based on the classic visual simultaneous localization and mapping (V-SLAM) framework. Our method propagates semantics on the scene graphs based on language priors and scene statistics to introduce semantic knowledge to the geometric frontiers.
  arXiv  Detail & Related papers  (2023-05-26T13:38:33Z)
• CroCo v2: Improved Cross-view Completion Pre-training for Stereo Matching and Optical Flow [22.161967080759993]
  Self-supervised pre-training methods have not yet delivered on dense geometric vision tasks such as stereo matching or optical flow. We build on the recent cross-view completion framework, a variation of masked image modeling that leverages a second view from the same scene. We show for the first time that state-of-the-art results on stereo matching and optical flow can be reached without using any classical task-specific techniques.
  arXiv  Detail & Related papers  (2022-11-18T18:18:53Z)
• CroCo: Self-Supervised Pre-training for 3D Vision Tasks by Cross-View Completion [20.121597331207276]
  Masked Image Modeling (MIM) has recently been established as a potent pre-training paradigm. In this paper we seek to learn representations that transfer well to a wide variety of 3D vision and lower-level geometric downstream tasks. Our experiments show that our pretext task leads to significantly improved performance for monocular 3D vision downstream tasks.
  arXiv  Detail & Related papers  (2022-10-19T16:50:36Z)
• Unpaired Referring Expression Grounding via Bidirectional Cross-Modal Matching [53.27673119360868]
  Referring expression grounding is an important and challenging task in computer vision. We propose a novel bidirectional cross-modal matching (BiCM) framework to address these challenges. Our framework outperforms previous works by 6.55% and 9.94% on two popular grounding datasets.
  arXiv  Detail & Related papers  (2022-01-18T01:13:19Z)
• Contrastive Object-level Pre-training with Spatial Noise Curriculum Learning [12.697842097171119]
  We present a curriculum learning mechanism that adaptively augments the generated regions, which allows the model to consistently acquire a useful learning signal. Our experiments show that our approach improves on the MoCo v2 baseline by a large margin on multiple object-level tasks when pre-training on multi-object scene image datasets.
  arXiv  Detail & Related papers  (2021-11-26T18:29:57Z)
• Warp Consistency for Unsupervised Learning of Dense Correspondences [116.56251250853488]
  Key challenge in learning dense correspondences is lack of ground-truth matches for real image pairs. We propose Warp Consistency, an unsupervised learning objective for dense correspondence regression. Our approach sets a new state-of-the-art on several challenging benchmarks, including MegaDepth, RobotCar and TSS.
  arXiv  Detail & Related papers  (2021-04-07T17:58:22Z)
• Tasks Integrated Networks: Joint Detection and Retrieval for Image Search [99.49021025124405]
  In many real-world searching scenarios (e.g., video surveillance), the objects are seldom accurately detected or annotated. We first introduce an end-to-end Integrated Net (I-Net), which has three merits. We further propose an improved I-Net, called DC-I-Net, which makes two new contributions.
  arXiv  Detail & Related papers  (2020-09-03T03:57:50Z)
• Improving Target-driven Visual Navigation with Attention on 3D Spatial Relationships [52.72020203771489]
  We investigate target-driven visual navigation using deep reinforcement learning (DRL) in 3D indoor scenes. Our proposed method combines visual features and 3D spatial representations to learn navigation policy. Our experiments, performed in the AI2-THOR, show that our model outperforms the baselines in both SR and SPL metrics.
  arXiv  Detail & Related papers  (2020-04-29T08:46:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.