Accelerating Reinforcement Learning with Learned Skill Priors
- URL: http://arxiv.org/abs/2010.11944v1
- Date: Thu, 22 Oct 2020 17:59:51 GMT
- Title: Accelerating Reinforcement Learning with Learned Skill Priors
- Authors: Karl Pertsch, Youngwoon Lee, Joseph J. Lim
- Abstract summary: Most modern reinforcement learning approaches learn every task from scratch.
One approach for leveraging prior knowledge is to transfer skills learned on prior tasks to the new task.
We show that learned skill priors are essential for effective skill transfer from rich datasets.
- Score: 20.268358783821487
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Intelligent agents rely heavily on prior experience when learning a new task,
yet most modern reinforcement learning (RL) approaches learn every task from
scratch. One approach for leveraging prior knowledge is to transfer skills
learned on prior tasks to the new task. However, as the amount of prior
experience increases, the number of transferable skills grows too, making it
challenging to explore the full set of available skills during downstream
learning. Yet, intuitively, not all skills should be explored with equal
probability; for example information about the current state can hint which
skills are promising to explore. In this work, we propose to implement this
intuition by learning a prior over skills. We propose a deep latent variable
model that jointly learns an embedding space of skills and the skill prior from
offline agent experience. We then extend common maximum-entropy RL approaches
to use skill priors to guide downstream learning. We validate our approach,
SPiRL (Skill-Prior RL), on complex navigation and robotic manipulation tasks
and show that learned skill priors are essential for effective skill transfer
from rich datasets. Videos and code are available at https://clvrai.com/spirl.
Related papers
- Leveraging Skills from Unlabeled Prior Data for Efficient Online Exploration [54.8229698058649]
We study how unlabeled prior trajectory data can be leveraged to learn efficient exploration strategies.
Our method SUPE (Skills from Unlabeled Prior data for Exploration) demonstrates that a careful combination of these ideas compounds their benefits.
We empirically show that SUPE reliably outperforms prior strategies, successfully solving a suite of long-horizon, sparse-reward tasks.
arXiv Detail & Related papers (2024-10-23T17:58:45Z) - EXTRACT: Efficient Policy Learning by Extracting Transferable Robot Skills from Offline Data [22.471559284344462]
Most reinforcement learning (RL) methods focus on learning optimal policies over low-level action spaces.
While these methods can perform well in their training environments, they lack the flexibility to transfer to new tasks.
We demonstrate through experiments in sparse, image-based, robot manipulation environments that can more quickly learn new tasks than prior works.
arXiv Detail & Related papers (2024-06-25T17:50:03Z) - Bootstrap Your Own Skills: Learning to Solve New Tasks with Large
Language Model Guidance [66.615355754712]
BOSS learns to accomplish new tasks by performing "skill bootstrapping"
We demonstrate through experiments in realistic household environments that agents trained with our LLM-guided bootstrapping procedure outperform those trained with naive bootstrapping.
arXiv Detail & Related papers (2023-10-16T02:43:47Z) - Skill Reinforcement Learning and Planning for Open-World Long-Horizon
Tasks [31.084848672383185]
We study building multi-task agents in open-world environments.
We convert the multi-task learning problem into learning basic skills and planning over the skills.
Our method accomplishes 40 diverse Minecraft tasks, where many tasks require sequentially executing for more than 10 skills.
arXiv Detail & Related papers (2023-03-29T09:45:50Z) - Choreographer: Learning and Adapting Skills in Imagination [60.09911483010824]
We present Choreographer, a model-based agent that exploits its world model to learn and adapt skills in imagination.
Our method decouples the exploration and skill learning processes, being able to discover skills in the latent state space of the model.
Choreographer is able to learn skills both from offline data, and by collecting data simultaneously with an exploration policy.
arXiv Detail & Related papers (2022-11-23T23:31:14Z) - Residual Skill Policies: Learning an Adaptable Skill-based Action Space
for Reinforcement Learning for Robotics [18.546688182454236]
Skill-based reinforcement learning (RL) has emerged as a promising strategy to leverage prior knowledge for accelerated robot learning.
We propose accelerating exploration in the skill space using state-conditioned generative models.
We validate our approach across four challenging manipulation tasks, demonstrating our ability to learn across task variations.
arXiv Detail & Related papers (2022-11-04T02:42:17Z) - Learning and Retrieval from Prior Data for Skill-based Imitation
Learning [47.59794569496233]
We develop a skill-based imitation learning framework that extracts temporally extended sensorimotor skills from prior data.
We identify several key design choices that significantly improve performance on novel tasks.
arXiv Detail & Related papers (2022-10-20T17:34:59Z) - Hierarchical Kickstarting for Skill Transfer in Reinforcement Learning [27.69559938165733]
Practising and honing skills forms a fundamental component of how humans learn, yet artificial agents are rarely specifically trained to perform them.
We investigate how skills can be incorporated into the training of reinforcement learning (RL) agents in complex environments.
Our experiments show that learning with a prior knowledge of useful skills can significantly improve the performance of agents on complex problems.
arXiv Detail & Related papers (2022-07-23T19:23:29Z) - Hierarchical Skills for Efficient Exploration [70.62309286348057]
In reinforcement learning, pre-trained low-level skills have the potential to greatly facilitate exploration.
Prior knowledge of the downstream task is required to strike the right balance between generality (fine-grained control) and specificity (faster learning) in skill design.
We propose a hierarchical skill learning framework that acquires skills of varying complexity in an unsupervised manner.
arXiv Detail & Related papers (2021-10-20T22:29:32Z) - Parrot: Data-Driven Behavioral Priors for Reinforcement Learning [79.32403825036792]
We propose a method for pre-training behavioral priors that can capture complex input-output relationships observed in successful trials.
We show how this learned prior can be used for rapidly learning new tasks without impeding the RL agent's ability to try out novel behaviors.
arXiv Detail & Related papers (2020-11-19T18:47:40Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.