ASPiRe:Adaptive Skill Priors for Reinforcement Learning
- URL: http://arxiv.org/abs/2209.15205v1
- Date: Fri, 30 Sep 2022 03:22:27 GMT
- Title: ASPiRe:Adaptive Skill Priors for Reinforcement Learning
- Authors: Mengda Xu, Manuela Veloso, Shuran Song
- Abstract summary: ASPiRe (Adaptive Skill Prior for RL) is a new approach to accelerate reinforcement learning.
Our framework learns a library of different distinction skill priors from a collection of specialized datasets.
Our experiments demonstrate that ASPiRe can significantly accelerate the learning of new downstream tasks.
- Score: 28.376277797807706
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We introduce ASPiRe (Adaptive Skill Prior for RL), a new approach that
leverages prior experience to accelerate reinforcement learning. Unlike
existing methods that learn a single skill prior from a large and diverse
dataset, our framework learns a library of different distinction skill priors
(i.e., behavior priors) from a collection of specialized datasets, and learns
how to combine them to solve a new task. This formulation allows the algorithm
to acquire a set of specialized skill priors that are more reusable for
downstream tasks; however, it also brings up additional challenges of how to
effectively combine these unstructured sets of skill priors to form a new prior
for new tasks. Specifically, it requires the agent not only to identify which
skill prior(s) to use but also how to combine them (either sequentially or
concurrently) to form a new prior. To achieve this goal, ASPiRe includes
Adaptive Weight Module (AWM) that learns to infer an adaptive weight assignment
between different skill priors and uses them to guide policy learning for
downstream tasks via weighted Kullback-Leibler divergences. Our experiments
demonstrate that ASPiRe can significantly accelerate the learning of new
downstream tasks in the presence of multiple priors and show improvement on
competitive baselines.
Related papers
- Skill Expansion and Composition in Parameter Space [17.016614374151747]
Parametric Skill Expansion and Composition (PSEC) is a new framework designed to iteratively evolve the agents' capabilities.
PSEC exhibits superior capacity to leverage prior knowledge to efficiently tackle new challenges.
arXiv Detail & Related papers (2025-02-09T15:22:38Z) - LW2G: Learning Whether to Grow for Prompt-based Continual Learning [15.766350352592331]
Recent Prompt-based Continual Learning (PCL) has achieved remarkable performance with Pre-Trained Models (PTMs)
We propose a plug-in module in the former stage to textbfLearn Whether to Grow (LW2G) based on the disparities between tasks.
Inspired by Gradient Projection Continual Learning, our LW2G develops a metric called Hinder Forward Capability (HFC) to measure the hindrance imposed on learning new tasks.
arXiv Detail & Related papers (2024-09-27T15:55:13Z) - Self-paced Weight Consolidation for Continual Learning [39.27729549041708]
Continual learning algorithms are popular in preventing catastrophic forgetting in sequential task learning settings.
We propose a self-paced Weight Consolidation (spWC) framework to attain continual learning.
arXiv Detail & Related papers (2023-07-20T13:07:41Z) - Learning Options via Compression [62.55893046218824]
We propose a new objective that combines the maximum likelihood objective with a penalty on the description length of the skills.
Our objective learns skills that solve downstream tasks in fewer samples compared to skills learned from only maximizing likelihood.
arXiv Detail & Related papers (2022-12-08T22:34:59Z) - Effective Adaptation in Multi-Task Co-Training for Unified Autonomous
Driving [103.745551954983]
In this paper, we investigate the transfer performance of various types of self-supervised methods, including MoCo and SimCLR, on three downstream tasks.
We find that their performances are sub-optimal or even lag far behind the single-task baseline.
We propose a simple yet effective pretrain-adapt-finetune paradigm for general multi-task training.
arXiv Detail & Related papers (2022-09-19T12:15:31Z) - Meta Reinforcement Learning with Successor Feature Based Context [51.35452583759734]
We propose a novel meta-RL approach that achieves competitive performance comparing to existing meta-RL algorithms.
Our method does not only learn high-quality policies for multiple tasks simultaneously but also can quickly adapt to new tasks with a small amount of training.
arXiv Detail & Related papers (2022-07-29T14:52:47Z) - Hierarchical Skills for Efficient Exploration [70.62309286348057]
In reinforcement learning, pre-trained low-level skills have the potential to greatly facilitate exploration.
Prior knowledge of the downstream task is required to strike the right balance between generality (fine-grained control) and specificity (faster learning) in skill design.
We propose a hierarchical skill learning framework that acquires skills of varying complexity in an unsupervised manner.
arXiv Detail & Related papers (2021-10-20T22:29:32Z) - The Information Geometry of Unsupervised Reinforcement Learning [133.20816939521941]
Unsupervised skill discovery is a class of algorithms that learn a set of policies without access to a reward function.
We show that unsupervised skill discovery algorithms do not learn skills that are optimal for every possible reward function.
arXiv Detail & Related papers (2021-10-06T13:08:36Z) - Parrot: Data-Driven Behavioral Priors for Reinforcement Learning [79.32403825036792]
We propose a method for pre-training behavioral priors that can capture complex input-output relationships observed in successful trials.
We show how this learned prior can be used for rapidly learning new tasks without impeding the RL agent's ability to try out novel behaviors.
arXiv Detail & Related papers (2020-11-19T18:47:40Z) - Accelerating Reinforcement Learning with Learned Skill Priors [20.268358783821487]
Most modern reinforcement learning approaches learn every task from scratch.
One approach for leveraging prior knowledge is to transfer skills learned on prior tasks to the new task.
We show that learned skill priors are essential for effective skill transfer from rich datasets.
arXiv Detail & Related papers (2020-10-22T17:59:51Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.