Seeing is Believing? Enhancing Vision-Language Navigation using Visual Perturbations

• URL: http://arxiv.org/abs/2409.05552v1
• Date: Mon, 9 Sep 2024 12:17:38 GMT
• Title: Seeing is Believing? Enhancing Vision-Language Navigation using Visual Perturbations
• Authors: Xuesong Zhang, Jia Li, Yunbo Xu, Zhenzhen Hu, Richang Hong,
• Abstract summary: We present a versatile Multi-Branch Architecture (MBA) aimed at exploring and exploiting diverse visual inputs. Specifically, we introduce three distinct visual variants: ground-truth depth images, visual inputs integrated with incongruent views, and those infused with random noise. Experiments conducted on three VLN benchmarks (R2R, REVERIE, SOON) demonstrate that our proposed method equals or even surpasses state-of-the-art results.
• Score: 41.5875455113941
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Autonomous navigation for an embodied agent guided by natural language instructions remains a formidable challenge in vision-and-language navigation (VLN). Despite remarkable recent progress in learning fine-grained and multifarious visual representations, the tendency to overfit to the training environments leads to unsatisfactory generalization performance. In this work, we present a versatile Multi-Branch Architecture (MBA) aimed at exploring and exploiting diverse visual inputs. Specifically, we introduce three distinct visual variants: ground-truth depth images, visual inputs integrated with incongruent views, and those infused with random noise to enrich the diversity of visual input representation and prevent overfitting to the original RGB observations. To adaptively fuse these varied inputs, the proposed MBA extend a base agent model into a multi-branch variant, where each branch processes a different visual input. Surprisingly, even random noise can further enhance navigation performance in unseen environments. Extensive experiments conducted on three VLN benchmarks (R2R, REVERIE, SOON) demonstrate that our proposed method equals or even surpasses state-of-the-art results. The source code will be publicly available.

Related papers

• UnitedVLN: Generalizable Gaussian Splatting for Continuous Vision-Language Navigation [71.97405667493477]
  We introduce a novel, generalizable 3DGS-based pre-training paradigm, called UnitedVLN. It enables agents to better explore future environments by unitedly rendering high-fidelity 360 visual images and semantic features. UnitedVLN outperforms state-of-the-art methods on existing VLN-CE benchmarks.
  arXiv  Detail & Related papers  (2024-11-25T02:44:59Z)
• Fine-Grained Alignment in Vision-and-Language Navigation through Bayesian Optimization [20.608059199982094]
  This paper addresses the challenge of fine-grained alignment in Vision-and-Language Navigation (VLN) tasks. Current approaches use contrastive learning to align language with visual trajectory sequences. We introduce a novel Bayesian Optimization-based adversarial optimization framework for creating fine-grained contrastive vision samples.
  arXiv  Detail & Related papers  (2024-11-22T09:12:02Z)
• Multi-Granularity Language-Guided Multi-Object Tracking [95.91263758294154]
  We propose a new multi-object tracking framework, named LG-MOT, that explicitly leverages language information at different levels of granularity. At inference, our LG-MOT uses the standard visual features without relying on annotated language descriptions. Our LG-MOT achieves an absolute gain of 2.2% in terms of target object association (IDF1 score) compared to the baseline using only visual features.
  arXiv  Detail & Related papers  (2024-06-07T11:18:40Z)
• Divert More Attention to Vision-Language Object Tracking [87.31882921111048]
  We argue that the lack of large-scale vision-language annotated videos and ineffective vision-language interaction learning motivate us to design more effective vision-language representation for tracking. Particularly, in this paper, we first propose a general attribute annotation strategy to decorate videos in six popular tracking benchmarks, which contributes a large-scale vision-language tracking database with more than 23,000 videos. We then introduce a novel framework to improve tracking by learning a unified-adaptive VL representation, where the cores are the proposed asymmetric architecture search and modality mixer (ModaMixer)
  arXiv  Detail & Related papers  (2023-07-19T15:22:06Z)
• GeoVLN: Learning Geometry-Enhanced Visual Representation with Slot Attention for Vision-and-Language Navigation [52.65506307440127]
  We propose GeoVLN, which learns Geometry-enhanced visual representation based on slot attention for robust Visual-and-Language Navigation. We employ V&L BERT to learn a cross-modal representation that incorporate both language and vision informations.
  arXiv  Detail & Related papers  (2023-05-26T17:15:22Z)
• TriPINet: Tripartite Progressive Integration Network for Image Manipulation Localization [3.7359400978194675]
  We propose a tripartite progressive integration network (TriPINet) for end-to-end image manipulation localization. We develop a guided cross-modality dual-attention (gCMDA) module to fuse different types of forged clues. Extensive experiments are conducted to compare our method with state-of-the-art image forensics approaches.
  arXiv  Detail & Related papers  (2022-12-25T02:27:58Z)
• Unsupervised Multimodal Neural Machine Translation with Pseudo Visual Pivoting [105.5303416210736]
  Unsupervised machine translation (MT) has recently achieved impressive results with monolingual corpora only. It is still challenging to associate source-target sentences in the latent space. As people speak different languages biologically share similar visual systems, the potential of achieving better alignment through visual content is promising.
  arXiv  Detail & Related papers  (2020-05-06T20:11:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.