Adversarial Environment Generation for Learning to Navigate the Web

• URL: http://arxiv.org/abs/2103.01991v1
• Date: Tue, 2 Mar 2021 19:19:30 GMT
• Title: Adversarial Environment Generation for Learning to Navigate the Web
• Authors: Izzeddin Gur, Natasha Jaques, Kevin Malta, Manoj Tiwari, Honglak Lee, Aleksandra Faust
• Abstract summary: One of the bottlenecks of training web navigation agents is providing a learnable curriculum of training environments. We propose using Adversarial Environment Generation (AEG) to generate challenging web environments in which to train reinforcement learning (RL) agents. We show that the navigator agent trained with our proposed Flexible b-PAIRED technique significantly outperforms competitive automatic curriculum generation baselines.
• Score: 107.99759923626242
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Learning to autonomously navigate the web is a difficult sequential decision making task. The state and action spaces are large and combinatorial in nature, and websites are dynamic environments consisting of several pages. One of the bottlenecks of training web navigation agents is providing a learnable curriculum of training environments that can cover the large variety of real-world websites. Therefore, we propose using Adversarial Environment Generation (AEG) to generate challenging web environments in which to train reinforcement learning (RL) agents. We provide a new benchmarking environment, gMiniWoB, which enables an RL adversary to use compositional primitives to learn to generate arbitrarily complex websites. To train the adversary, we propose a new technique for maximizing regret using the difference in the scores obtained by a pair of navigator agents. Our results show that our approach significantly outperforms prior methods for minimax regret AEG. The regret objective trains the adversary to design a curriculum of environments that are "just-the-right-challenge" for the navigator agents; our results show that over time, the adversary learns to generate increasingly complex web navigation tasks. The navigator agents trained with our technique learn to complete challenging, high-dimensional web navigation tasks, such as form filling, booking a flight etc. We show that the navigator agent trained with our proposed Flexible b-PAIRED technique significantly outperforms competitive automatic curriculum generation baselines -- including a state-of-the-art RL web navigation approach -- on a set of challenging unseen test environments, and achieves more than 80% success rate on some tasks.

Related papers

• A Role of Environmental Complexity on Representation Learning in Deep Reinforcement Learning Agents [3.7314353481448337]
  We developed a simulated navigation environment to train deep reinforcement learning agents. We modulated the frequency of exposure to a shortcut and navigation cue, leading to the development of artificial agents with differing abilities. We examined the encoded representations in artificial neural networks driving these agents, revealing intricate dynamics in representation learning.
  arXiv  Detail & Related papers  (2024-07-03T18:27:26Z)
• Aquatic Navigation: A Challenging Benchmark for Deep Reinforcement Learning [53.3760591018817]
  We propose a new benchmarking environment for aquatic navigation using recent advances in the integration between game engines and Deep Reinforcement Learning. Specifically, we focus on PPO, one of the most widely accepted algorithms, and we propose advanced training techniques. Our empirical evaluation shows that a well-designed combination of these ingredients can achieve promising results.
  arXiv  Detail & Related papers  (2024-05-30T23:20:23Z)
• Visual-Language Navigation Pretraining via Prompt-based Environmental Self-exploration [83.96729205383501]
  We introduce prompt-based learning to achieve fast adaptation for language embeddings. Our model can adapt to diverse vision-language navigation tasks, including VLN and REVERIE.
  arXiv  Detail & Related papers  (2022-03-08T11:01:24Z)
• Environment Generation for Zero-Shot Compositional Reinforcement Learning [105.35258025210862]
  Compositional Design of Environments (CoDE) trains a Generator agent to automatically build a series of compositional tasks tailored to the agent's current skill level. We learn to generate environments composed of multiple pages or rooms, and train RL agents capable of completing wide-range of complex tasks in those environments. CoDE yields 4x higher success rate than the strongest baseline, and demonstrates strong performance of real websites learned on 3500 primitive tasks.
  arXiv  Detail & Related papers  (2022-01-21T21:35:01Z)
• Human-Aware Robot Navigation via Reinforcement Learning with Hindsight Experience Replay and Curriculum Learning [28.045441768064215]
  Reinforcement learning approaches have shown superior ability in solving sequential decision making problems. In this work, we consider the task of training an RL agent without employing the demonstration data. We propose to incorporate the hindsight experience replay (HER) and curriculum learning (CL) techniques with RL to efficiently learn the optimal navigation policy in the dense crowd.
  arXiv  Detail & Related papers  (2021-10-09T13:18:11Z)
• ViNG: Learning Open-World Navigation with Visual Goals [82.84193221280216]
  We propose a learning-based navigation system for reaching visually indicated goals. We show that our system, which we call ViNG, outperforms previously-proposed methods for goal-conditioned reinforcement learning. We demonstrate ViNG on a number of real-world applications, such as last-mile delivery and warehouse inspection.
  arXiv  Detail & Related papers  (2020-12-17T18:22:32Z)
• Embodied Visual Navigation with Automatic Curriculum Learning in Real Environments [20.017277077448924]
  NavACL is a method of automatic curriculum learning tailored to the navigation task. Deep reinforcement learning agents trained using NavACL significantly outperform state-of-the-art agents trained with uniform sampling. Our agents can navigate through unknown cluttered indoor environments to semantically-specified targets using only RGB images.
  arXiv  Detail & Related papers  (2020-09-11T13:28:26Z)
• Active Visual Information Gathering for Vision-Language Navigation [115.40768457718325]
  Vision-language navigation (VLN) is the task of entailing an agent to carry out navigational instructions inside photo-realistic environments. One of the key challenges in VLN is how to conduct a robust navigation by mitigating the uncertainty caused by ambiguous instructions and insufficient observation of the environment. This work draws inspiration from human navigation behavior and endows an agent with an active information gathering ability for a more intelligent VLN policy.
  arXiv  Detail & Related papers  (2020-07-15T23:54:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.