Agent Spaces

• URL: http://arxiv.org/abs/2111.06005v1
• Date: Thu, 11 Nov 2021 01:12:17 GMT
• Title: Agent Spaces
• Authors: John C. Raisbeck, Matthew W. Allen, Hakho Lee
• Abstract summary: We define exploration as the act of modifying an agent to itself be explorative. We show that many important structures in Reinforcement Learning are well behaved under the topology induced by convergence in the agent space.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Exploration is one of the most important tasks in Reinforcement Learning, but it is not well-defined beyond finite problems in the Dynamic Programming paradigm (see Subsection 2.4). We provide a reinterpretation of exploration which can be applied to any online learning method. We come to this definition by approaching exploration from a new direction. After finding that concepts of exploration created to solve simple Markov decision processes with Dynamic Programming are no longer broadly applicable, we reexamine exploration. Instead of extending the ends of dynamic exploration procedures, we extend their means. That is, rather than repeatedly sampling every state-action pair possible in a process, we define the act of modifying an agent to itself be explorative. The resulting definition of exploration can be applied in infinite problems and non-dynamic learning methods, which the dynamic notion of exploration cannot tolerate. To understand the way that modifications of an agent affect learning, we describe a novel structure on the set of agents: a collection of distances (see footnote 7) $d_{a} \in A$, which represent the perspectives of each agent possible in the process. Using these distances, we define a topology and show that many important structures in Reinforcement Learning are well behaved under the topology induced by convergence in the agent space.

Related papers

• A Definition of Continual Reinforcement Learning [69.56273766737527]
  In a standard view of the reinforcement learning problem, an agent's goal is to efficiently identify a policy that maximizes long-term reward. Continual reinforcement learning refers to the setting in which the best agents never stop learning. We formalize the notion of agents that "never stop learning" through a new mathematical language for analyzing and cataloging agents.
  arXiv  Detail & Related papers  (2023-07-20T17:28:01Z)
• Deep Intrinsically Motivated Exploration in Continuous Control [0.0]
  In continuous systems, exploration is often performed through undirected strategies in which parameters of the networks or selected actions are perturbed by random noise. We adapt existing theories on animal motivational systems into the reinforcement learning paradigm and introduce a novel directed exploration strategy. Our framework extends to larger and more diverse state spaces, dramatically improves the baselines, and outperforms the undirected strategies significantly.
  arXiv  Detail & Related papers  (2022-10-01T14:52:16Z)
• Interesting Object, Curious Agent: Learning Task-Agnostic Exploration [44.18450799034677]
  In this paper, we propose a paradigm change in the formulation and evaluation of task-agnostic exploration. We show that our formulation is effective and provides the most consistent exploration across several training-testing environment pairs.
  arXiv  Detail & Related papers  (2021-11-25T15:17:32Z)
• Long-Term Exploration in Persistent MDPs [68.8204255655161]
  We propose an exploration method called Rollback-Explore (RbExplore) In this paper, we propose an exploration method called Rollback-Explore (RbExplore), which utilizes the concept of the persistent Markov decision process. We test our algorithm in the hard-exploration Prince of Persia game, without rewards and domain knowledge.
  arXiv  Detail & Related papers  (2021-09-21T13:47:04Z)
• Cooperative Exploration for Multi-Agent Deep Reinforcement Learning [127.4746863307944]
  We propose cooperative multi-agent exploration (CMAE) for deep reinforcement learning. The goal is selected from multiple projected state spaces via a normalized entropy-based technique. We demonstrate that CMAE consistently outperforms baselines on various tasks.
  arXiv  Detail & Related papers  (2021-07-23T20:06:32Z)
• MURAL: Meta-Learning Uncertainty-Aware Rewards for Outcome-Driven Reinforcement Learning [65.52675802289775]
  We show that an uncertainty aware classifier can solve challenging reinforcement learning problems. We propose a novel method for computing the normalized maximum likelihood (NML) distribution. We show that the resulting algorithm has a number of intriguing connections to both count-based exploration methods and prior algorithms for learning reward functions.
  arXiv  Detail & Related papers  (2021-07-15T08:19:57Z)
• Objective discovery of dominant dynamical processes with intelligible machine learning [0.0]
  We present a formal definition in which the identification of dynamical regimes is formulated as an optimization problem. We propose an unsupervised learning framework which eliminates the need for a priori knowledge and ad hoc definitions. Our method is a step towards unbiased data exploration that allows serendipitous discovery within dynamical systems.
  arXiv  Detail & Related papers  (2021-06-21T20:57:23Z)
• A Novel Approach to Curiosity and Explainable Reinforcement Learning via Interpretable Sub-Goals [0.0]
  Two key challenges within Reinforcement Learning involve improving (a) agent learning within environments with sparse extrinsic rewards and (b) the explainability of agent actions. We describe a curious subgoal focused agent to address both these challenges. We use a novel method for curiosity produced from a Generative Adrial Network (GAN) based model of environment transitions that is robust to environment transitions.
  arXiv  Detail & Related papers  (2021-04-14T05:21:13Z)
• Progressive growing of self-organized hierarchical representations for exploration [22.950651316748207]
  A central challenge is how to learn representations in order to progressively build a map of the discovered structures. We aim to build lasting representations and avoid catastrophic forgetting throughout the exploration process. Thirdly, we target representations that can structure the agent discoveries in a coarse-to-fine manner.
  arXiv  Detail & Related papers  (2020-05-13T15:24:42Z)
• Planning to Explore via Self-Supervised World Models [120.31359262226758]
  Plan2Explore is a self-supervised reinforcement learning agent. We present a new approach to self-supervised exploration and fast adaptation to new tasks. Without any training supervision or task-specific interaction, Plan2Explore outperforms prior self-supervised exploration methods.
  arXiv  Detail & Related papers  (2020-05-12T17:59:45Z)
• Meta Reinforcement Learning with Autonomous Inference of Subtask Dependencies [57.27944046925876]
  We propose and address a novel few-shot RL problem, where a task is characterized by a subtask graph. Instead of directly learning a meta-policy, we develop a Meta-learner with Subtask Graph Inference. Our experiment results on two grid-world domains and StarCraft II environments show that the proposed method is able to accurately infer the latent task parameter.
  arXiv  Detail & Related papers  (2020-01-01T17:34:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.