Mixing Human Demonstrations with Self-Exploration in Experience Replay for Deep Reinforcement Learning

• URL: http://arxiv.org/abs/2107.06840v1
• Date: Wed, 14 Jul 2021 16:55:30 GMT
• Title: Mixing Human Demonstrations with Self-Exploration in Experience Replay for Deep Reinforcement Learning
• Authors: Dylan Klein, Akansel Cosgun
• Abstract summary: We investigate the effect of using human demonstration data in the replay buffer for Deep Reinforcement Learning. Our results suggest that while the agents trained by pure self-exploration and pure demonstration had similar success rates, the pure demonstration model converged faster to solutions with less number of steps.
• Score: 2.8783296093434148
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We investigate the effect of using human demonstration data in the replay buffer for Deep Reinforcement Learning. We use a policy gradient method with a modified experience replay buffer where a human demonstration experience is sampled with a given probability. We analyze different ratios of using demonstration data in a task where an agent attempts to reach a goal while avoiding obstacles. Our results suggest that while the agents trained by pure self-exploration and pure demonstration had similar success rates, the pure demonstration model converged faster to solutions with less number of steps.

Related papers

• Inverse-RLignment: Inverse Reinforcement Learning from Demonstrations for LLM Alignment [62.05713042908654]
  We introduce Alignment from Demonstrations (AfD), a novel approach leveraging high-quality demonstration data to overcome these challenges. We formalize AfD within a sequential decision-making framework, highlighting its unique challenge of missing reward signals. Practically, we propose a computationally efficient algorithm that extrapolates over a tailored reward model for AfD.
  arXiv  Detail & Related papers  (2024-05-24T15:13:53Z)
• Zero-shot Imitation Policy via Search in Demonstration Dataset [0.16817021284806563]
  Behavioral cloning uses a dataset of demonstrations to learn a policy. We propose to use latent spaces of pre-trained foundation models to index a demonstration dataset. Our approach can effectively recover meaningful demonstrations and show human-like behavior of an agent in the Minecraft environment.
  arXiv  Detail & Related papers  (2024-01-29T18:38:29Z)
• Data Pruning via Moving-one-Sample-out [61.45441981346064]
  We propose a novel data-pruning approach called moving-one-sample-out (MoSo) MoSo aims to identify and remove the least informative samples from the training set. Experimental results demonstrate that MoSo effectively mitigates severe performance degradation at high pruning ratios.
  arXiv  Detail & Related papers  (2023-10-23T08:00:03Z)
• Accelerating exploration and representation learning with offline pre-training [52.6912479800592]
  We show that exploration and representation learning can be improved by separately learning two different models from a single offline dataset. We show that learning a state representation using noise-contrastive estimation and a model of auxiliary reward can significantly improve the sample efficiency on the challenging NetHack benchmark.
  arXiv  Detail & Related papers  (2023-03-31T18:03:30Z)
• Robust Imitation of a Few Demonstrations with a Backwards Model [3.8530020696501794]
  Behavior cloning of expert demonstrations can speed up learning optimal policies in a more sample-efficient way than reinforcement learning. We tackle this issue by extending the region of attraction around the demonstrations so that the agent can learn how to get back onto the demonstrated trajectories if it veers off-course. With optimal or near-optimal demonstrations, the learned policy will be both optimal and robust to deviations, with a wider region of attraction.
  arXiv  Detail & Related papers  (2022-10-17T18:02:19Z)
• Evaluating the Effectiveness of Corrective Demonstrations and a Low-Cost Sensor for Dexterous Manipulation [0.5669790037378094]
  Imitation learning is a promising approach to help robots acquire dexterous manipulation capabilities. We investigate characteristics of such additional demonstrations and their impact on performance. We show that inexpensive vision-based sensors, such as LeapMotion, can be used to dramatically reduce the cost of providing demonstrations.
  arXiv  Detail & Related papers  (2022-04-15T19:55:46Z)
• SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning [168.89470249446023]
  We present SURF, a semi-supervised reward learning framework that utilizes a large amount of unlabeled samples with data augmentation. In order to leverage unlabeled samples for reward learning, we infer pseudo-labels of the unlabeled samples based on the confidence of the preference predictor. Our experiments demonstrate that our approach significantly improves the feedback-efficiency of the preference-based method on a variety of locomotion and robotic manipulation tasks.
  arXiv  Detail & Related papers  (2022-03-18T16:50:38Z)
• Sampling Attacks: Amplification of Membership Inference Attacks by Repeated Queries [74.59376038272661]
  We introduce sampling attack, a novel membership inference technique that unlike other standard membership adversaries is able to work under severe restriction of no access to scores of the victim model. We show that a victim model that only publishes the labels is still susceptible to sampling attacks and the adversary can recover up to 100% of its performance. For defense, we choose differential privacy in the form of gradient perturbation during the training of the victim model as well as output perturbation at prediction time.
  arXiv  Detail & Related papers  (2020-09-01T12:54:54Z)
• Forgetful Experience Replay in Hierarchical Reinforcement Learning from Demonstrations [55.41644538483948]
  In this paper, we propose a combination of approaches that allow the agent to use low-quality demonstrations in complex vision-based environments. Our proposed goal-oriented structuring of replay buffer allows the agent to automatically highlight sub-goals for solving complex hierarchical tasks in demonstrations. The solution based on our algorithm beats all the solutions for the famous MineRL competition and allows the agent to mine a diamond in the Minecraft environment.
  arXiv  Detail & Related papers  (2020-06-17T15:38:40Z)
• Reinforcement Learning with Supervision from Noisy Demonstrations [38.00968774243178]
  We propose a novel framework to adaptively learn the policy by jointly interacting with the environment and exploiting the expert demonstrations. Experimental results in various environments with multiple popular reinforcement learning algorithms show that the proposed approach can learn robustly with noisy demonstrations.
  arXiv  Detail & Related papers  (2020-06-14T06:03:06Z)
• Sample Efficient Reinforcement Learning through Learning from Demonstrations in Minecraft [4.3952888284140785]
  We show how human demonstrations can improve final performance of agents on the Minecraft minigame ObtainDiamond with only 8M frames of environment interaction. Our solution placed 3rd in the NeurIPS MineRL Competition for Sample-Efficient Reinforcement Learning.
  arXiv  Detail & Related papers  (2020-03-12T23:46:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.