Hierarchical Kickstarting for Skill Transfer in Reinforcement Learning
- URL: http://arxiv.org/abs/2207.11584v1
- Date: Sat, 23 Jul 2022 19:23:29 GMT
- Title: Hierarchical Kickstarting for Skill Transfer in Reinforcement Learning
- Authors: Michael Matthews, Mikayel Samvelyan, Jack Parker-Holder, Edward
Grefenstette, Tim Rockt\"aschel
- Abstract summary: Practising and honing skills forms a fundamental component of how humans learn, yet artificial agents are rarely specifically trained to perform them.
We investigate how skills can be incorporated into the training of reinforcement learning (RL) agents in complex environments.
Our experiments show that learning with a prior knowledge of useful skills can significantly improve the performance of agents on complex problems.
- Score: 27.69559938165733
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Practising and honing skills forms a fundamental component of how humans
learn, yet artificial agents are rarely specifically trained to perform them.
Instead, they are usually trained end-to-end, with the hope being that useful
skills will be implicitly learned in order to maximise discounted return of
some extrinsic reward function. In this paper, we investigate how skills can be
incorporated into the training of reinforcement learning (RL) agents in complex
environments with large state-action spaces and sparse rewards. To this end, we
created SkillHack, a benchmark of tasks and associated skills based on the game
of NetHack. We evaluate a number of baselines on this benchmark, as well as our
own novel skill-based method Hierarchical Kickstarting (HKS), which is shown to
outperform all other evaluated methods. Our experiments show that learning with
a prior knowledge of useful skills can significantly improve the performance of
agents on complex problems. We ultimately argue that utilising predefined
skills provides a useful inductive bias for RL problems, especially those with
large state-action spaces and sparse rewards.
Related papers
- EXTRACT: Efficient Policy Learning by Extracting Transferable Robot Skills from Offline Data [22.471559284344462]
Most reinforcement learning (RL) methods focus on learning optimal policies over low-level action spaces.
While these methods can perform well in their training environments, they lack the flexibility to transfer to new tasks.
We demonstrate through experiments in sparse, image-based, robot manipulation environments that can more quickly learn new tasks than prior works.
arXiv Detail & Related papers (2024-06-25T17:50:03Z) - When Do Skills Help Reinforcement Learning? A Theoretical Analysis of Temporal Abstractions [12.74839237274274]
Skills are temporal abstractions intended to improve reinforcement learning (RL) performance.
We show theoretically and empirically that RL performance gain from skills is worse in environments where solutions to states are less compressible.
We hope our findings can guide research on automatic skill discovery and help RL practitioners better decide when and how to use skills.
arXiv Detail & Related papers (2024-06-12T06:01:42Z) - Acquiring Diverse Skills using Curriculum Reinforcement Learning with Mixture of Experts [58.220879689376744]
Reinforcement learning (RL) is a powerful approach for acquiring a good-performing policy.
We propose textbfDiverse textbfSkill textbfLearning (Di-SkilL) for learning diverse skills.
We show on challenging robot simulation tasks that Di-SkilL can learn diverse and performant skills.
arXiv Detail & Related papers (2024-03-11T17:49:18Z) - Residual Skill Policies: Learning an Adaptable Skill-based Action Space
for Reinforcement Learning for Robotics [18.546688182454236]
Skill-based reinforcement learning (RL) has emerged as a promising strategy to leverage prior knowledge for accelerated robot learning.
We propose accelerating exploration in the skill space using state-conditioned generative models.
We validate our approach across four challenging manipulation tasks, demonstrating our ability to learn across task variations.
arXiv Detail & Related papers (2022-11-04T02:42:17Z) - Skill-Based Reinforcement Learning with Intrinsic Reward Matching [77.34726150561087]
We present Intrinsic Reward Matching (IRM), which unifies task-agnostic skill pretraining and task-aware finetuning.
IRM enables us to utilize pretrained skills far more effectively than previous skill selection methods.
arXiv Detail & Related papers (2022-10-14T00:04:49Z) - Basis for Intentions: Efficient Inverse Reinforcement Learning using
Past Experience [89.30876995059168]
inverse reinforcement learning (IRL) -- inferring the reward function of an agent from observing its behavior.
This paper addresses the problem of IRL -- inferring the reward function of an agent from observing its behavior.
arXiv Detail & Related papers (2022-08-09T17:29:49Z) - Rethinking Learning Dynamics in RL using Adversarial Networks [79.56118674435844]
We present a learning mechanism for reinforcement learning of closely related skills parameterized via a skill embedding space.
The main contribution of our work is to formulate an adversarial training regime for reinforcement learning with the help of entropy-regularized policy gradient formulation.
arXiv Detail & Related papers (2022-01-27T19:51:09Z) - Hierarchical Skills for Efficient Exploration [70.62309286348057]
In reinforcement learning, pre-trained low-level skills have the potential to greatly facilitate exploration.
Prior knowledge of the downstream task is required to strike the right balance between generality (fine-grained control) and specificity (faster learning) in skill design.
We propose a hierarchical skill learning framework that acquires skills of varying complexity in an unsupervised manner.
arXiv Detail & Related papers (2021-10-20T22:29:32Z) - Accelerating Reinforcement Learning with Learned Skill Priors [20.268358783821487]
Most modern reinforcement learning approaches learn every task from scratch.
One approach for leveraging prior knowledge is to transfer skills learned on prior tasks to the new task.
We show that learned skill priors are essential for effective skill transfer from rich datasets.
arXiv Detail & Related papers (2020-10-22T17:59:51Z) - Emergent Real-World Robotic Skills via Unsupervised Off-Policy
Reinforcement Learning [81.12201426668894]
We develop efficient reinforcement learning methods that acquire diverse skills without any reward function, and then repurpose these skills for downstream tasks.
We show that our proposed algorithm provides substantial improvement in learning efficiency, making reward-free real-world training feasible.
We also demonstrate that the learned skills can be composed using model predictive control for goal-oriented navigation, without any additional training.
arXiv Detail & Related papers (2020-04-27T17:38:53Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.