Diagnosing the Environment Bias in Vision-and-Language Navigation

• URL: http://arxiv.org/abs/2005.03086v1
• Date: Wed, 6 May 2020 19:24:33 GMT
• Title: Diagnosing the Environment Bias in Vision-and-Language Navigation
• Authors: Yubo Zhang, Hao Tan, Mohit Bansal
• Abstract summary: Vision-and-Language Navigation (VLN) requires an agent to follow natural-language instructions, explore the given environments, and reach the desired target locations. Recent works that study VLN observe a significant performance drop when tested on unseen environments, indicating that the neural agent models are highly biased towards training environments. In this work, we design novel diagnosis experiments via environment re-splitting and feature replacement, looking into possible reasons for this environment bias.
• Score: 102.02103792590076
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-and-Language Navigation (VLN) requires an agent to follow natural-language instructions, explore the given environments, and reach the desired target locations. These step-by-step navigational instructions are crucial when the agent is navigating new environments about which it has no prior knowledge. Most recent works that study VLN observe a significant performance drop when tested on unseen environments (i.e., environments not used in training), indicating that the neural agent models are highly biased towards training environments. Although this issue is considered as one of the major challenges in VLN research, it is still under-studied and needs a clearer explanation. In this work, we design novel diagnosis experiments via environment re-splitting and feature replacement, looking into possible reasons for this environment bias. We observe that neither the language nor the underlying navigational graph, but the low-level visual appearance conveyed by ResNet features directly affects the agent model and contributes to this environment bias in results. According to this observation, we explore several kinds of semantic representations that contain less low-level visual information, hence the agent learned with these features could be better generalized to unseen testing environments. Without modifying the baseline agent model and its training method, our explored semantic features significantly decrease the performance gaps between seen and unseen on multiple datasets (i.e. R2R, R4R, and CVDN) and achieve competitive unseen results to previous state-of-the-art models. Our code and features are available at: https://github.com/zhangybzbo/EnvBiasVLN

Related papers

• UnitedVLN: Generalizable Gaussian Splatting for Continuous Vision-Language Navigation [71.97405667493477]
  We introduce a novel, generalizable 3DGS-based pre-training paradigm, called UnitedVLN. It enables agents to better explore future environments by unitedly rendering high-fidelity 360 visual images and semantic features. UnitedVLN outperforms state-of-the-art methods on existing VLN-CE benchmarks.
  arXiv  Detail & Related papers  (2024-11-25T02:44:59Z)
• Interpretable Brain-Inspired Representations Improve RL Performance on Visual Navigation Tasks [0.0]
  We show how the method of slow feature analysis (SFA) overcomes both limitations by generating interpretable representations of visual data. We employ SFA in a modern reinforcement learning context, analyse and compare representations and illustrate where hierarchical SFA can outperform other feature extractors on navigation tasks.
  arXiv  Detail & Related papers  (2024-02-19T11:35:01Z)
• Anticipating the Unseen Discrepancy for Vision and Language Navigation [63.399180481818405]
  Vision-Language Navigation requires the agent to follow natural language instructions to reach a specific target. The large discrepancy between seen and unseen environments makes it challenging for the agent to generalize well. We propose Unseen Discrepancy Anticipating Vision and Language Navigation (DAVIS) that learns to generalize to unseen environments via encouraging test-time visual consistency.
  arXiv  Detail & Related papers  (2022-09-10T19:04:40Z)
• What do navigation agents learn about their environment? [39.74076893981299]
  We introduce the Interpretability System for Embodied agEnts (iSEE) for Point Goal and Object Goal navigation agents. We use iSEE to probe the dynamic representations produced by these agents for the presence of information about the agent as well as the environment.
  arXiv  Detail & Related papers  (2022-06-17T01:33:43Z)
• Glimpse-Attend-and-Explore: Self-Attention for Active Visual Exploration [47.01485765231528]
  Active visual exploration aims to assist an agent with a limited field of view to understand its environment based on partial observations. We propose the Glimpse-Attend-and-Explore model which employs self-attention to guide the visual exploration instead of task-specific uncertainty maps. Our model provides encouraging results while being less dependent on dataset bias in driving the exploration.
  arXiv  Detail & Related papers  (2021-08-26T11:41:03Z)
• Vision-Language Navigation with Random Environmental Mixup [112.94609558723518]
  Vision-language Navigation (VLN) tasks require an agent to navigate step-by-step while perceiving the visual observations and comprehending a natural language instruction. Previous works have proposed various data augmentation methods to reduce data bias. We propose the Random Environmental Mixup (REM) method, which generates cross-connected house scenes as augmented data via mixuping environment.
  arXiv  Detail & Related papers  (2021-06-15T04:34:26Z)
• Diagnosing Vision-and-Language Navigation: What Really Matters [61.72935815656582]
  Vision-and-language navigation (VLN) is a multimodal task where an agent follows natural language instructions and navigates in visual environments. Recent studies witness a slow-down in the performance improvements in both indoor and outdoor VLN tasks. In this work, we conduct a series of diagnostic experiments to unveil agents' focus during navigation.
  arXiv  Detail & Related papers  (2021-03-30T17:59:07Z)
• Environment-agnostic Multitask Learning for Natural Language Grounded Navigation [88.69873520186017]
  We introduce a multitask navigation model that can be seamlessly trained on Vision-Language Navigation (VLN) and Navigation from Dialog History (NDH) tasks. Experiments show that environment-agnostic multitask learning significantly reduces the performance gap between seen and unseen environments.
  arXiv  Detail & Related papers  (2020-03-01T09:06:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.