Optimal Options for Multi-Task Reinforcement Learning Under Time Constraints

• URL: http://arxiv.org/abs/2001.01620v1
• Date: Mon, 6 Jan 2020 15:08:46 GMT
• Title: Optimal Options for Multi-Task Reinforcement Learning Under Time Constraints
• Authors: Manuel Del Verme, Bruno Castro da Silva, Gianluca Baldassarre
• Abstract summary: Reinforcement learning can benefit from the use of options as a way of encoding recurring behaviours and to foster exploration. We investigate some of the conditions that influence optimality of options, in settings where agents have a limited time budget for learning each task. We show that the discovered options significantly differ depending on factors such as the available learning time budget and that the found options outperform popular option-generations.
• Score: 0.6445605125467573
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement learning can greatly benefit from the use of options as a way of encoding recurring behaviours and to foster exploration. An important open problem is how can an agent autonomously learn useful options when solving particular distributions of related tasks. We investigate some of the conditions that influence optimality of options, in settings where agents have a limited time budget for learning each task and the task distribution might involve problems with different levels of similarity. We directly search for optimal option sets and show that the discovered options significantly differ depending on factors such as the available learning time budget and that the found options outperform popular option-generation heuristics.

Related papers

• Finding Optimal Diverse Feature Sets with Alternative Feature Selection [0.0]
  We introduce alternative feature selection and formalize it as an optimization problem. In particular, we define alternatives via constraints and enable users to control the number and dissimilarity of alternatives. We show that a constant-factor approximation exists under certain conditions and propose corresponding search methods.
  arXiv  Detail & Related papers  (2023-07-21T14:23:41Z)
• The Paradox of Choice: Using Attention in Hierarchical Reinforcement Learning [59.777127897688594]
  We present an online, model-free algorithm to learn affordances that can be used to further learn subgoal options. We investigate the role of hard versus soft attention in training data collection, abstract value learning in long-horizon tasks, and handling a growing number of choices.
  arXiv  Detail & Related papers  (2022-01-24T13:18:02Z)
• Attention Option-Critic [56.50123642237106]
  We propose an attention-based extension to the option-critic framework. We show that this leads to behaviorally diverse options which are also capable of state abstraction. We also demonstrate the more efficient, interpretable, and reusable nature of the learned options in comparison with option-critic.
  arXiv  Detail & Related papers  (2022-01-07T18:44:28Z)
• Temporal Abstraction in Reinforcement Learning with the Successor Representation [65.69658154078007]
  We argue that the successor representation (SR) can be seen as a natural substrate for the discovery and use of temporal abstractions. We show how the SR can be used to discover options that facilitate either temporally-extended exploration or planning.
  arXiv  Detail & Related papers  (2021-10-12T05:07:43Z)
• Adversarial Option-Aware Hierarchical Imitation Learning [89.92994158193237]
  We propose Option-GAIL, a novel method to learn skills at long horizon. The key idea of Option-GAIL is modeling the task hierarchy by options and train the policy via generative adversarial optimization. Experiments show that Option-GAIL outperforms other counterparts consistently across a variety of tasks.
  arXiv  Detail & Related papers  (2021-06-10T06:42:05Z)
• Discovery of Options via Meta-Learned Subgoals [59.2160583043938]
  Temporal abstractions in the form of options have been shown to help reinforcement learning (RL) agents learn faster. We introduce a novel meta-gradient approach for discovering useful options in multi-task RL environments.
  arXiv  Detail & Related papers  (2021-02-12T19:50:40Z)
• Diversity-Enriched Option-Critic [47.82697599507171]
  We show that our proposed method is capable of learning options end-to-end on several discrete and continuous control tasks. Our approach generates robust, reusable, reliable and interpretable options, in contrast to option-critic.
  arXiv  Detail & Related papers  (2020-11-04T22:12:54Z)
• Learning Diverse Options via InfoMax Termination Critic [0.0]
  We consider the problem of autonomously learning reusable temporally extended actions, or options, in reinforcement learning. Motivated by the recent success of mutual information based skill learning, we hypothesize that more diverse options are more reusable. We propose a method for learning gradient of options by maximizing MI between options and corresponding state transitions.
  arXiv  Detail & Related papers  (2020-10-06T14:21:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.