A Simple Approach to Continual Learning by Transferring Skill Parameters
- URL: http://arxiv.org/abs/2110.10255v1
- Date: Tue, 19 Oct 2021 20:44:20 GMT
- Title: A Simple Approach to Continual Learning by Transferring Skill Parameters
- Authors: K.R. Zentner, Ryan Julian, Ujjwal Puri, Yulun Zhang, Gaurav S.
Sukhatme
- Abstract summary: We show how to continually acquire robotic manipulation skills without forgetting, and using far fewer samples than needed to train them from scratch.
Given an appropriate curriculum, we show how to continually acquire robotic manipulation skills without forgetting, and using far fewer samples than needed to train them from scratch.
- Score: 25.705923249267055
- License: http://creativecommons.org/licenses/by-sa/4.0/
- Abstract: In order to be effective general purpose machines in real world environments,
robots not only will need to adapt their existing manipulation skills to new
circumstances, they will need to acquire entirely new skills on-the-fly. A
great promise of continual learning is to endow robots with this ability, by
using their accumulated knowledge and experience from prior skills. We take a
fresh look at this problem, by considering a setting in which the robot is
limited to storing that knowledge and experience only in the form of learned
skill policies. We show that storing skill policies, careful pre-training, and
appropriately choosing when to transfer those skill policies is sufficient to
build a continual learner in the context of robotic manipulation. We analyze
which conditions are needed to transfer skills in the challenging Meta-World
simulation benchmark. Using this analysis, we introduce a pair-wise metric
relating skills that allows us to predict the effectiveness of skill transfer
between tasks, and use it to reduce the problem of continual learning to
curriculum selection. Given an appropriate curriculum, we show how to
continually acquire robotic manipulation skills without forgetting, and using
far fewer samples than needed to train them from scratch.
Related papers
- Unsupervised Skill Discovery for Robotic Manipulation through Automatic Task Generation [17.222197596599685]
We propose a Skill Learning approach that discovers composable behaviors by solving a large number of autonomously generated tasks.
Our method learns skills allowing the robot to consistently and robustly interact with objects in its environment.
The learned skills can be used to solve a set of unseen manipulation tasks, in simulation as well as on a real robotic platform.
arXiv Detail & Related papers (2024-10-07T09:19:13Z) - Task-free Lifelong Robot Learning with Retrieval-based Weighted Local Adaptation [17.215730187681952]
We store a subset of data from previous tasks and utilize it in two manners: leveraging experience replay to retain learned skills and applying a novel Retrieval-based Local Adaptation technique to restore relevant knowledge.
We also incorporate a selective weighting mechanism to focus on the most "forgotten" skill segment, ensuring effective knowledge restoration.
arXiv Detail & Related papers (2024-10-03T21:11:42Z) - Practice Makes Perfect: Planning to Learn Skill Parameter Policies [34.51008914846429]
In this work, we focus on the active learning problem of choosing which skills to practice to maximize expected future task success.
We propose that the robot should estimate the competence of each skill, extrapolate the competence, and situate the skill in the task distribution through competence-aware planning.
This approach is implemented within a fully autonomous system where the robot repeatedly plans, practices, and learns without any environment resets.
arXiv Detail & Related papers (2024-02-22T23:58:26Z) - Choreographer: Learning and Adapting Skills in Imagination [60.09911483010824]
We present Choreographer, a model-based agent that exploits its world model to learn and adapt skills in imagination.
Our method decouples the exploration and skill learning processes, being able to discover skills in the latent state space of the model.
Choreographer is able to learn skills both from offline data, and by collecting data simultaneously with an exploration policy.
arXiv Detail & Related papers (2022-11-23T23:31:14Z) - Learning and Retrieval from Prior Data for Skill-based Imitation
Learning [47.59794569496233]
We develop a skill-based imitation learning framework that extracts temporally extended sensorimotor skills from prior data.
We identify several key design choices that significantly improve performance on novel tasks.
arXiv Detail & Related papers (2022-10-20T17:34:59Z) - Hierarchical Skills for Efficient Exploration [70.62309286348057]
In reinforcement learning, pre-trained low-level skills have the potential to greatly facilitate exploration.
Prior knowledge of the downstream task is required to strike the right balance between generality (fine-grained control) and specificity (faster learning) in skill design.
We propose a hierarchical skill learning framework that acquires skills of varying complexity in an unsupervised manner.
arXiv Detail & Related papers (2021-10-20T22:29:32Z) - Discovering Generalizable Skills via Automated Generation of Diverse
Tasks [82.16392072211337]
We propose a method to discover generalizable skills via automated generation of a diverse set of tasks.
As opposed to prior work on unsupervised discovery of skills, our method pairs each skill with a unique task produced by a trainable task generator.
A task discriminator defined on the robot behaviors in the generated tasks is jointly trained to estimate the evidence lower bound of the diversity objective.
The learned skills can then be composed in a hierarchical reinforcement learning algorithm to solve unseen target tasks.
arXiv Detail & Related papers (2021-06-26T03:41:51Z) - Actionable Models: Unsupervised Offline Reinforcement Learning of
Robotic Skills [93.12417203541948]
We propose the objective of learning a functional understanding of the environment by learning to reach any goal state in a given dataset.
We find that our method can operate on high-dimensional camera images and learn a variety of skills on real robots that generalize to previously unseen scenes and objects.
arXiv Detail & Related papers (2021-04-15T20:10:11Z) - Learning and Sequencing of Object-Centric Manipulation Skills for
Industrial Tasks [16.308562047398542]
We propose a rapid robot skill-sequencing algorithm, where the skills are encoded by object-centric hidden semi-Markov models.
The learned skill models can encode multimodal (temporal and spatial) trajectory distributions.
We demonstrate this approach on a 7 DoF robot arm for industrial assembly tasks.
arXiv Detail & Related papers (2020-08-24T14:20:05Z) - Never Stop Learning: The Effectiveness of Fine-Tuning in Robotic
Reinforcement Learning [109.77163932886413]
We show how to adapt vision-based robotic manipulation policies to new variations by fine-tuning via off-policy reinforcement learning.
This adaptation uses less than 0.2% of the data necessary to learn the task from scratch.
We find that our approach of adapting pre-trained policies leads to substantial performance gains over the course of fine-tuning.
arXiv Detail & Related papers (2020-04-21T17:57:04Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.