RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environments

• URL: http://arxiv.org/abs/2002.12292v2
• Date: Sat, 29 Feb 2020 16:12:58 GMT
• Title: RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environments
• Authors: Roberta Raileanu and Tim Rockt\"aschel
• Abstract summary: We propose a novel type of intrinsic reward which encourages the agent to take actions that lead to significant changes in its learned state representation. We evaluate our method on multiple challenging procedurally-generated tasks in MiniGrid.
• Score: 15.736899098702972
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Exploration in sparse reward environments remains one of the key challenges of model-free reinforcement learning. Instead of solely relying on extrinsic rewards provided by the environment, many state-of-the-art methods use intrinsic rewards to encourage exploration. However, we show that existing methods fall short in procedurally-generated environments where an agent is unlikely to visit a state more than once. We propose a novel type of intrinsic reward which encourages the agent to take actions that lead to significant changes in its learned state representation. We evaluate our method on multiple challenging procedurally-generated tasks in MiniGrid, as well as on tasks with high-dimensional observations used in prior work. Our experiments demonstrate that this approach is more sample efficient than existing exploration methods, particularly for procedurally-generated MiniGrid environments. Furthermore, we analyze the learned behavior as well as the intrinsic reward received by our agent. In contrast to previous approaches, our intrinsic reward does not diminish during the course of training and it rewards the agent substantially more for interacting with objects that it can control.

Related papers

• Random Latent Exploration for Deep Reinforcement Learning [71.88709402926415]
  This paper introduces a new exploration technique called Random Latent Exploration (RLE) RLE combines the strengths of bonus-based and noise-based (two popular approaches for effective exploration in deep RL) exploration strategies. We evaluate it on the challenging Atari and IsaacGym benchmarks and show that RLE exhibits higher overall scores across all the tasks than other approaches.
  arXiv  Detail & Related papers  (2024-07-18T17:55:22Z)
• DEIR: Efficient and Robust Exploration through Discriminative-Model-Based Episodic Intrinsic Rewards [2.09711130126031]
  Exploration is a fundamental aspect of reinforcement learning (RL), and its effectiveness is a deciding factor in the performance of RL algorithms. Recent studies have shown the effectiveness of encouraging exploration with intrinsic rewards estimated from novelties in observations. We propose DEIR, a novel method in which we theoretically derive an intrinsic reward with a conditional mutual information term.
  arXiv  Detail & Related papers  (2023-04-21T06:39:38Z)
• Self-supervised network distillation: an effective approach to exploration in sparse reward environments [0.0]
  Reinforcement learning can train an agent to behave in an environment according to a predesigned reward function. The solution to such a problem may be to equip the agent with an intrinsic motivation that will provide informed exploration. We present Self-supervised Network Distillation (SND), a class of intrinsic motivation algorithms based on the distillation error as a novelty indicator.
  arXiv  Detail & Related papers  (2023-02-22T18:58:09Z)
• Self-Supervised Exploration via Latent Bayesian Surprise [4.088019409160893]
  In this work, we propose a curiosity-based bonus as intrinsic reward for Reinforcement Learning. We extensively evaluate our model by measuring the agent's performance in terms of environment exploration. Our model is cheap and empirically shows state-of-the-art performance on several problems.
  arXiv  Detail & Related papers  (2021-04-15T14:40:16Z)
• Rank the Episodes: A Simple Approach for Exploration in Procedurally-Generated Environments [66.80667987347151]
  Methods based on intrinsic rewards often fall short in procedurally-generated environments. We introduce RAPID, a simple yet effective episode-level exploration method for procedurally-generated environments. We demonstrate our method on several procedurally-generated MiniGrid environments, a first-person-view 3D Maze navigation task from MiniWorld, and several sparse MuJoCo tasks.
  arXiv  Detail & Related papers  (2021-01-20T14:22:01Z)
• BeBold: Exploration Beyond the Boundary of Explored Regions [66.88415950549556]
  In this paper, we propose the regulated difference of inverse visitation counts as a simple but effective criterion for intrinsic reward (IR) The criterion helps the agent explore Beyond the Boundary of explored regions and mitigates common issues in count-based methods, such as short-sightedness and detachment. The resulting method, BeBold, solves the 12 most challenging procedurally-generated tasks in MiniGrid with just 120M environment steps, without any curriculum learning.
  arXiv  Detail & Related papers  (2020-12-15T21:26:54Z)
• Demonstration-efficient Inverse Reinforcement Learning in Procedurally Generated Environments [137.86426963572214]
  Inverse Reinforcement Learning can extrapolate reward functions from expert demonstrations. We show that our approach, DE-AIRL, is demonstration-efficient and still able to extrapolate reward functions which generalize to the fully procedural domain.
  arXiv  Detail & Related papers  (2020-12-04T11:18:02Z)
• Variational Dynamic for Self-Supervised Exploration in Deep Reinforcement Learning [12.76337275628074]
  In this work, we propose a variational dynamic model based on the conditional variational inference to model the multimodality andgenerativeity. We derive an upper bound of the negative log-likelihood of the environmental transition and use such an upper bound as the intrinsic reward for exploration. Our method outperforms several state-of-the-art environment model-based exploration approaches.
  arXiv  Detail & Related papers  (2020-10-17T09:54:51Z)
• Planning to Explore via Self-Supervised World Models [120.31359262226758]
  Plan2Explore is a self-supervised reinforcement learning agent. We present a new approach to self-supervised exploration and fast adaptation to new tasks. Without any training supervision or task-specific interaction, Plan2Explore outperforms prior self-supervised exploration methods.
  arXiv  Detail & Related papers  (2020-05-12T17:59:45Z)
• Never Give Up: Learning Directed Exploration Strategies [63.19616370038824]
  We propose a reinforcement learning agent to solve hard exploration games by learning a range of directed exploratory policies. We construct an episodic memory-based intrinsic reward using k-nearest neighbors over the agent's recent experience to train the directed exploratory policies. A self-supervised inverse dynamics model is used to train the embeddings of the nearest neighbour lookup, biasing the novelty signal towards what the agent can control.
  arXiv  Detail & Related papers  (2020-02-14T13:57:22Z)
• Long-Term Visitation Value for Deep Exploration in Sparse Reward Reinforcement Learning [34.38011902445557]
  Reinforcement learning with sparse rewards is still an open challenge. We present a novel approach that plans exploration actions far into the future by using a long-term visitation count. Contrary to existing methods which use models of reward and dynamics, our approach is off-policy and model-free.
  arXiv  Detail & Related papers  (2020-01-01T01:01:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.