Adaptive Policy Transfer in Reinforcement Learning
- URL: http://arxiv.org/abs/2105.04699v1
- Date: Mon, 10 May 2021 22:42:03 GMT
- Title: Adaptive Policy Transfer in Reinforcement Learning
- Authors: Girish Joshi, Girish Chowdhary
- Abstract summary: We introduce a principled mechanism that can "Adapt-to-Learn", that is adapt the source policy to learn to solve a target task.
We show that the presented method learns to seamlessly combine learning from adaptation and exploration and leads to a robust policy transfer algorithm.
- Score: 9.594432031144715
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Efficient and robust policy transfer remains a key challenge for
reinforcement learning to become viable for real-wold robotics. Policy transfer
through warm initialization, imitation, or interacting over a large set of
agents with randomized instances, have been commonly applied to solve a variety
of Reinforcement Learning tasks. However, this seems far from how skill
transfer happens in the biological world: Humans and animals are able to
quickly adapt the learned behaviors between similar tasks and learn new skills
when presented with new situations. Here we seek to answer the question: Will
learning to combine adaptation and exploration lead to a more efficient
transfer of policies between domains? We introduce a principled mechanism that
can "Adapt-to-Learn", that is adapt the source policy to learn to solve a
target task with significant transition differences and uncertainties. We show
that the presented method learns to seamlessly combine learning from adaptation
and exploration and leads to a robust policy transfer algorithm with
significantly reduced sample complexity in transferring skills between related
tasks.
Related papers
- Diversity for Contingency: Learning Diverse Behaviors for Efficient
Adaptation and Transfer [0.0]
We propose a simple method for discovering all possible solutions of a given task.
Unlike prior methods, our approach does not require learning additional models for novelty detection.
arXiv Detail & Related papers (2023-10-11T13:39:35Z) - IOB: Integrating Optimization Transfer and Behavior Transfer for
Multi-Policy Reuse [50.90781542323258]
Reinforcement learning (RL) agents can transfer knowledge from source policies to a related target task.
Previous methods introduce additional components, such as hierarchical policies or estimations of source policies' value functions.
We propose a novel transfer RL method that selects the source policy without training extra components.
arXiv Detail & Related papers (2023-08-14T09:22:35Z) - Learning Multi-Task Transferable Rewards via Variational Inverse
Reinforcement Learning [10.782043595405831]
We extend an empowerment-based regularization technique to situations with multiple tasks based on the framework of a generative adversarial network.
Under the multitask environments with unknown dynamics, we focus on learning a reward and policy from unlabeled expert examples.
Our proposed method derives the variational lower bound of the situational mutual information to optimize it.
arXiv Detail & Related papers (2022-06-19T22:32:41Z) - Multi-Agent Policy Transfer via Task Relationship Modeling [28.421365805638953]
We try to discover and exploit common structures among tasks for more efficient transfer.
We propose to learn effect-based task representations as a common space of tasks, using an alternatively fixed training scheme.
As a result, the proposed method can help transfer learned cooperation knowledge to new tasks after training on a few source tasks.
arXiv Detail & Related papers (2022-03-09T01:49:21Z) - Rethinking Learning Dynamics in RL using Adversarial Networks [79.56118674435844]
We present a learning mechanism for reinforcement learning of closely related skills parameterized via a skill embedding space.
The main contribution of our work is to formulate an adversarial training regime for reinforcement learning with the help of entropy-regularized policy gradient formulation.
arXiv Detail & Related papers (2022-01-27T19:51:09Z) - Transferability in Deep Learning: A Survey [80.67296873915176]
The ability to acquire and reuse knowledge is known as transferability in deep learning.
We present this survey to connect different isolated areas in deep learning with their relation to transferability.
We implement a benchmark and an open-source library, enabling a fair evaluation of deep learning methods in terms of transferability.
arXiv Detail & Related papers (2022-01-15T15:03:17Z) - Coverage as a Principle for Discovering Transferable Behavior in
Reinforcement Learning [16.12658895065585]
We argue that representation alone is not enough for efficient transfer in challenging domains and explore how to transfer knowledge through behavior.
The behavior of pre-trained policies may be used for solving the task at hand (exploitation) or for collecting useful data to solve the problem (exploration)
arXiv Detail & Related papers (2021-02-24T16:51:02Z) - Importance Weighted Policy Learning and Adaptation [89.46467771037054]
We study a complementary approach which is conceptually simple, general, modular and built on top of recent improvements in off-policy learning.
The framework is inspired by ideas from the probabilistic inference literature and combines robust off-policy learning with a behavior prior.
Our approach achieves competitive adaptation performance on hold-out tasks compared to meta reinforcement learning baselines and can scale to complex sparse-reward scenarios.
arXiv Detail & Related papers (2020-09-10T14:16:58Z) - Meta-Reinforcement Learning Robust to Distributional Shift via Model
Identification and Experience Relabeling [126.69933134648541]
We present a meta-reinforcement learning algorithm that is both efficient and extrapolates well when faced with out-of-distribution tasks at test time.
Our method is based on a simple insight: we recognize that dynamics models can be adapted efficiently and consistently with off-policy data.
arXiv Detail & Related papers (2020-06-12T13:34:46Z) - Efficient Deep Reinforcement Learning via Adaptive Policy Transfer [50.51637231309424]
Policy Transfer Framework (PTF) is proposed to accelerate Reinforcement Learning (RL)
Our framework learns when and which source policy is the best to reuse for the target policy and when to terminate it.
Experimental results show it significantly accelerates the learning process and surpasses state-of-the-art policy transfer methods.
arXiv Detail & Related papers (2020-02-19T07:30:57Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.