Residual Reinforcement Learning from Demonstrations
- URL: http://arxiv.org/abs/2106.08050v1
- Date: Tue, 15 Jun 2021 11:16:49 GMT
- Title: Residual Reinforcement Learning from Demonstrations
- Authors: Minttu Alakuijala (WILLOW, Thoth), Gabriel Dulac-Arnold, Julien Mairal
(Thoth), Jean Ponce (WILLOW), Cordelia Schmid
- Abstract summary: Residual reinforcement learning (RL) has been proposed as a way to solve challenging robotic tasks by adapting control actions from a conventional feedback controller to maximize a reward signal.
We extend the residual formulation to learn from visual inputs and sparse rewards using demonstrations.
Our experimental evaluation on simulated manipulation tasks on a 6-DoF UR5 arm and a 28-DoF dexterous hand demonstrates that residual RL from demonstrations is able to generalize to unseen environment conditions more flexibly than either behavioral cloning or RL fine-tuning.
- Score: 51.56457466788513
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Residual reinforcement learning (RL) has been proposed as a way to solve
challenging robotic tasks by adapting control actions from a conventional
feedback controller to maximize a reward signal. We extend the residual
formulation to learn from visual inputs and sparse rewards using
demonstrations. Learning from images, proprioceptive inputs and a sparse
task-completion reward relaxes the requirement of accessing full state
features, such as object and target positions. In addition, replacing the base
controller with a policy learned from demonstrations removes the dependency on
a hand-engineered controller in favour of a dataset of demonstrations, which
can be provided by non-experts. Our experimental evaluation on simulated
manipulation tasks on a 6-DoF UR5 arm and a 28-DoF dexterous hand demonstrates
that residual RL from demonstrations is able to generalize to unseen
environment conditions more flexibly than either behavioral cloning or RL
fine-tuning, and is capable of solving high-dimensional, sparse-reward tasks
out of reach for RL from scratch.
Related papers
- World Models Increase Autonomy in Reinforcement Learning [6.151562278670799]
Reinforcement learning (RL) is an appealing paradigm for training intelligent agents.
MoReFree agent adapts two key mechanisms, exploration and policy learning, to handle reset-free tasks.
It exhibits superior data-efficiency across various reset-free tasks without access to environmental reward or demonstrations.
arXiv Detail & Related papers (2024-08-19T08:56:00Z) - Offline Reinforcement Learning with Imputed Rewards [8.856568375969848]
We propose a Reward Model that can estimate the reward signal from a very limited sample of environment transitions annotated with rewards.
Our results show that, using only 1% of reward-labeled transitions from the original datasets, our learned reward model is able to impute rewards for the remaining 99% of the transitions.
arXiv Detail & Related papers (2024-07-15T15:53:13Z) - DEAR: Disentangled Environment and Agent Representations for Reinforcement Learning without Reconstruction [4.813546138483559]
Reinforcement Learning (RL) algorithms can learn robotic control tasks from visual observations, but they often require a large amount of data.
In this paper, we explore how the agent's knowledge of its shape can improve the sample efficiency of visual RL methods.
We propose a novel method, Disentangled Environment and Agent Representations, that uses the segmentation mask of the agent as supervision.
arXiv Detail & Related papers (2024-06-30T09:15:21Z) - Inverse-RLignment: Inverse Reinforcement Learning from Demonstrations for LLM Alignment [62.05713042908654]
We introduce Alignment from Demonstrations (AfD), a novel approach leveraging high-quality demonstration data to overcome these challenges.
We formalize AfD within a sequential decision-making framework, highlighting its unique challenge of missing reward signals.
Practically, we propose a computationally efficient algorithm that extrapolates over a tailored reward model for AfD.
arXiv Detail & Related papers (2024-05-24T15:13:53Z) - Sequential Action-Induced Invariant Representation for Reinforcement
Learning [1.2046159151610263]
How to accurately learn task-relevant state representations from high-dimensional observations with visual distractions is a challenging problem in visual reinforcement learning.
We propose a Sequential Action-induced invariant Representation (SAR) method, in which the encoder is optimized by an auxiliary learner to only preserve the components that follow the control signals of sequential actions.
arXiv Detail & Related papers (2023-09-22T05:31:55Z) - D-Shape: Demonstration-Shaped Reinforcement Learning via Goal
Conditioning [48.57484755946714]
D-Shape is a new method for combining imitation learning (IL) and reinforcement learning (RL)
This paper introduces D-Shape, a new method for combining IL and RL that uses ideas from reward shaping and goal-conditioned RL to resolve the above conflict.
We experimentally validate D-Shape in sparse-reward gridworld domains, showing that it both improves over RL in terms of sample efficiency and converges consistently to the optimal policy.
arXiv Detail & Related papers (2022-10-26T02:28:32Z) - Exploratory State Representation Learning [63.942632088208505]
We propose a new approach called XSRL (eXploratory State Representation Learning) to solve the problems of exploration and SRL in parallel.
On one hand, it jointly learns compact state representations and a state transition estimator which is used to remove unexploitable information from the representations.
On the other hand, it continuously trains an inverse model, and adds to the prediction error of this model a $k$-step learning progress bonus to form the objective of a discovery policy.
arXiv Detail & Related papers (2021-09-28T10:11:07Z) - Return-Based Contrastive Representation Learning for Reinforcement
Learning [126.7440353288838]
We propose a novel auxiliary task that forces the learnt representations to discriminate state-action pairs with different returns.
Our algorithm outperforms strong baselines on complex tasks in Atari games and DeepMind Control suite.
arXiv Detail & Related papers (2021-02-22T13:04:18Z) - Forgetful Experience Replay in Hierarchical Reinforcement Learning from
Demonstrations [55.41644538483948]
In this paper, we propose a combination of approaches that allow the agent to use low-quality demonstrations in complex vision-based environments.
Our proposed goal-oriented structuring of replay buffer allows the agent to automatically highlight sub-goals for solving complex hierarchical tasks in demonstrations.
The solution based on our algorithm beats all the solutions for the famous MineRL competition and allows the agent to mine a diamond in the Minecraft environment.
arXiv Detail & Related papers (2020-06-17T15:38:40Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.