GRIMGEP: Learning Progress for Robust Goal Sampling in Visual Deep Reinforcement Learning

• URL: http://arxiv.org/abs/2008.04388v2
• Date: Wed, 7 Jul 2021 11:54:09 GMT
• Title: GRIMGEP: Learning Progress for Robust Goal Sampling in Visual Deep Reinforcement Learning
• Authors: Grgur Kova\v{c}, Adrien Laversanne-Finot, Pierre-Yves Oudeyer
• Abstract summary: We propose a framework that allows agents to autonomously identify and ignore noisy distracting regions. Our framework can be combined with any state-of-the-art novelty seeking goal exploration approaches.
• Score: 21.661530291654692
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Designing agents, capable of learning autonomously a wide range of skills is critical in order to increase the scope of reinforcement learning. It will both increase the diversity of learned skills and reduce the burden of manually designing reward functions for each skill. Self-supervised agents, setting their own goals, and trying to maximize the diversity of those goals have shown great promise towards this end. However, a currently known limitation of agents trying to maximize the diversity of sampled goals is that they tend to get attracted to noise or more generally to parts of the environments that cannot be controlled (distractors). When agents have access to predefined goal features or expert knowledge, absolute Learning Progress (ALP) provides a way to distinguish between regions that can be controlled and those that cannot. However, those methods often fall short when the agents are only provided with raw sensory inputs such as images. In this work we extend those concepts to unsupervised image-based goal exploration. We propose a framework that allows agents to autonomously identify and ignore noisy distracting regions while searching for novelty in the learnable regions to both improve overall performance and avoid catastrophic forgetting. Our framework can be combined with any state-of-the-art novelty seeking goal exploration approaches. We construct a rich 3D image based environment with distractors. Experiments on this environment show that agents using our framework successfully identify interesting regions of the environment, resulting in drastically improved performances. The source code is available at https://sites.google.com/view/grimgep.

Related papers

• Choreographer: Learning and Adapting Skills in Imagination [60.09911483010824]
  We present Choreographer, a model-based agent that exploits its world model to learn and adapt skills in imagination. Our method decouples the exploration and skill learning processes, being able to discover skills in the latent state space of the model. Choreographer is able to learn skills both from offline data, and by collecting data simultaneously with an exploration policy.
  arXiv  Detail & Related papers  (2022-11-23T23:31:14Z)
• Stein Variational Goal Generation for adaptive Exploration in Multi-Goal Reinforcement Learning [18.62133925594957]
  In multi-goal Reinforcement Learning, an agent can share experience between related training tasks, resulting in better generalization at test time. In this work we propose Stein Variational Goal Generation (SVGG), which samples goals of intermediate difficulty for the agent. The distribution of goals is modeled with particles that are attracted in areas of appropriate difficulty using Stein Variational Gradient Descent.
  arXiv  Detail & Related papers  (2022-06-14T10:03:17Z)
• Deep Hierarchical Planning from Pixels [86.14687388689204]
  Director is a method for learning hierarchical behaviors directly from pixels by planning inside the latent space of a learned world model. Despite operating in latent space, the decisions are interpretable because the world model can decode goals into images for visualization. Director also learns successful behaviors across a wide range of environments, including visual control, Atari games, and DMLab levels.
  arXiv  Detail & Related papers  (2022-06-08T18:20:15Z)
• Stochastic Coherence Over Attention Trajectory For Continuous Learning In Video Streams [64.82800502603138]
  This paper proposes a novel neural-network-based approach to progressively and autonomously develop pixel-wise representations in a video stream. The proposed method is based on a human-like attention mechanism that allows the agent to learn by observing what is moving in the attended locations. Our experiments leverage 3D virtual environments and they show that the proposed agents can learn to distinguish objects just by observing the video stream.
  arXiv  Detail & Related papers  (2022-04-26T09:52:31Z)
• Learning Object-Centered Autotelic Behaviors with Graph Neural Networks [10.149376933379036]
  Humans have access to a handful of previously learned skills, which they rapidly adapt to new situations. In artificial intelligence, autotelic agents, which are intrinsically motivated to represent and set their own goals, exhibit promising skill adaptation capabilities. We study different implementations of autotelic agents using four types of Graph Neural Networks policy representations and two types of goal spaces.
  arXiv  Detail & Related papers  (2022-04-11T14:19:04Z)
• Direct then Diffuse: Incremental Unsupervised Skill Discovery for State Covering and Goal Reaching [98.25207998996066]
  We build on the mutual information framework for skill discovery and introduce UPSIDE to address the coverage-directedness trade-off. We illustrate in several navigation and control environments how the skills learned by UPSIDE solve sparse-reward downstream tasks better than existing baselines.
  arXiv  Detail & Related papers  (2021-10-27T14:22:19Z)
• Discovering and Achieving Goals via World Models [61.95437238374288]
  We introduce Latent Explorer Achiever (LEXA), a unified solution to this problem. LEXA learns a world model from image inputs and uses it to train an explorer and an achiever policy from imagined rollouts. After the unsupervised phase, LEXA solves tasks specified as goal images zero-shot without any additional learning.
  arXiv  Detail & Related papers  (2021-10-18T17:59:58Z)
• Domain-Robust Visual Imitation Learning with Mutual Information Constraints [0.0]
  We introduce a new algorithm called Disentangling Generative Adversarial Imitation Learning (DisentanGAIL) Our algorithm enables autonomous agents to learn directly from high dimensional observations of an expert performing a task.
  arXiv  Detail & Related papers  (2021-03-08T21:18:58Z)
• Action and Perception as Divergence Minimization [43.75550755678525]
  Action Perception Divergence is an approach for categorizing the space of possible objective functions for embodied agents. We show a spectrum that reaches from narrow to general objectives. These agents use perception to align their beliefs with the world and use actions to align the world with their beliefs.
  arXiv  Detail & Related papers  (2020-09-03T16:52:46Z)
• Automatic Curriculum Learning through Value Disagreement [95.19299356298876]
  Continually solving new, unsolved tasks is the key to learning diverse behaviors. In the multi-task domain, where an agent needs to reach multiple goals, the choice of training goals can largely affect sample efficiency. We propose setting up an automatic curriculum for goals that the agent needs to solve. We evaluate our method across 13 multi-goal robotic tasks and 5 navigation tasks, and demonstrate performance gains over current state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-06-17T03:58:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.