Choreographer: Learning and Adapting Skills in Imagination

• URL: http://arxiv.org/abs/2211.13350v2
• Date: Fri, 19 Jan 2024 17:33:36 GMT
• Title: Choreographer: Learning and Adapting Skills in Imagination
• Authors: Pietro Mazzaglia, Tim Verbelen, Bart Dhoedt, Alexandre Lacoste, Sai Rajeswar
• Abstract summary: We present Choreographer, a model-based agent that exploits its world model to learn and adapt skills in imagination. Our method decouples the exploration and skill learning processes, being able to discover skills in the latent state space of the model. Choreographer is able to learn skills both from offline data, and by collecting data simultaneously with an exploration policy.
• Score: 60.09911483010824
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Unsupervised skill learning aims to learn a rich repertoire of behaviors without external supervision, providing artificial agents with the ability to control and influence the environment. However, without appropriate knowledge and exploration, skills may provide control only over a restricted area of the environment, limiting their applicability. Furthermore, it is unclear how to leverage the learned skill behaviors for adapting to downstream tasks in a data-efficient manner. We present Choreographer, a model-based agent that exploits its world model to learn and adapt skills in imagination. Our method decouples the exploration and skill learning processes, being able to discover skills in the latent state space of the model. During adaptation, the agent uses a meta-controller to evaluate and adapt the learned skills efficiently by deploying them in parallel in imagination. Choreographer is able to learn skills both from offline data, and by collecting data simultaneously with an exploration policy. The skills can be used to effectively adapt to downstream tasks, as we show in the URL benchmark, where we outperform previous approaches from both pixels and states inputs. The learned skills also explore the environment thoroughly, finding sparse rewards more frequently, as shown in goal-reaching tasks from the DMC Suite and Meta-World. Website and code: https://skillchoreographer.github.io/

Related papers

• Unsupervised Skill Discovery for Robotic Manipulation through Automatic Task Generation [17.222197596599685]
  We propose a Skill Learning approach that discovers composable behaviors by solving a large number of autonomously generated tasks. Our method learns skills allowing the robot to consistently and robustly interact with objects in its environment. The learned skills can be used to solve a set of unseen manipulation tasks, in simulation as well as on a real robotic platform.
  arXiv  Detail & Related papers  (2024-10-07T09:19:13Z)
• Residual Skill Policies: Learning an Adaptable Skill-based Action Space for Reinforcement Learning for Robotics [18.546688182454236]
  Skill-based reinforcement learning (RL) has emerged as a promising strategy to leverage prior knowledge for accelerated robot learning. We propose accelerating exploration in the skill space using state-conditioned generative models. We validate our approach across four challenging manipulation tasks, demonstrating our ability to learn across task variations.
  arXiv  Detail & Related papers  (2022-11-04T02:42:17Z)
• Learning and Retrieval from Prior Data for Skill-based Imitation Learning [47.59794569496233]
  We develop a skill-based imitation learning framework that extracts temporally extended sensorimotor skills from prior data. We identify several key design choices that significantly improve performance on novel tasks.
  arXiv  Detail & Related papers  (2022-10-20T17:34:59Z)
• One After Another: Learning Incremental Skills for a Changing World [19.051800747558794]
  We propose a new framework for skill discovery, where skills are learned one after another in an incremental fashion. We demonstrate experimentally that in both evolving and static environments, incremental skills significantly outperform current state-of-the-art skill discovery methods.
  arXiv  Detail & Related papers  (2022-03-21T17:55:21Z)
• Hierarchical Skills for Efficient Exploration [70.62309286348057]
  In reinforcement learning, pre-trained low-level skills have the potential to greatly facilitate exploration. Prior knowledge of the downstream task is required to strike the right balance between generality (fine-grained control) and specificity (faster learning) in skill design. We propose a hierarchical skill learning framework that acquires skills of varying complexity in an unsupervised manner.
  arXiv  Detail & Related papers  (2021-10-20T22:29:32Z)
• Curious Representation Learning for Embodied Intelligence [81.21764276106924]
  Self-supervised representation learning has achieved remarkable success in recent years. Yet to build truly intelligent agents, we must construct representation learning algorithms that can learn from environments. We propose a framework, curious representation learning, which jointly learns a reinforcement learning policy and a visual representation model.
  arXiv  Detail & Related papers  (2021-05-03T17:59:20Z)
• Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills [93.12417203541948]
  We propose the objective of learning a functional understanding of the environment by learning to reach any goal state in a given dataset. We find that our method can operate on high-dimensional camera images and learn a variety of skills on real robots that generalize to previously unseen scenes and objects.
  arXiv  Detail & Related papers  (2021-04-15T20:10:11Z)
• ELSIM: End-to-end learning of reusable skills through intrinsic motivation [0.0]
  We present a novel reinforcement learning architecture which hierarchically learns and represents self-generated skills in an end-to-end way. With this architecture, an agent focuses only on task-rewarded skills while keeping the learning process of skills bottom-up.
  arXiv  Detail & Related papers  (2020-06-23T11:20:46Z)
• Emergent Real-World Robotic Skills via Unsupervised Off-Policy Reinforcement Learning [81.12201426668894]
  We develop efficient reinforcement learning methods that acquire diverse skills without any reward function, and then repurpose these skills for downstream tasks. We show that our proposed algorithm provides substantial improvement in learning efficiency, making reward-free real-world training feasible. We also demonstrate that the learned skills can be composed using model predictive control for goal-oriented navigation, without any additional training.
  arXiv  Detail & Related papers  (2020-04-27T17:38:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.