Offline Reinforcement Learning Hands-On

• URL: http://arxiv.org/abs/2011.14379v1
• Date: Sun, 29 Nov 2020 14:45:02 GMT
• Title: Offline Reinforcement Learning Hands-On
• Authors: Louis Monier, Jakub Kmec, Alexandre Laterre, Thomas Pierrot, Valentin Courgeau, Olivier Sigaud and Karim Beguir
• Abstract summary: offline RL aims to turn large datasets into powerful decision-making engines without any online interactions with the environment. This work aims to reflect upon these efforts from a practitioner viewpoint. We experimentally validate that diversity and high-return examples in the data are crucial to the success of offline RL.
• Score: 60.36729294485601
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Offline Reinforcement Learning (RL) aims to turn large datasets into powerful decision-making engines without any online interactions with the environment. This great promise has motivated a large amount of research that hopes to replicate the success RL has experienced in simulation settings. This work ambitions to reflect upon these efforts from a practitioner viewpoint. We start by discussing the dataset properties that we hypothesise can characterise the type of offline methods that will be the most successful. We then verify these claims through a set of experiments and designed datasets generated from environments with both discrete and continuous action spaces. We experimentally validate that diversity and high-return examples in the data are crucial to the success of offline RL and show that behavioural cloning remains a strong contender compared to its contemporaries. Overall, this work stands as a tutorial to help people build their intuition on today's offline RL methods and their applicability.

Related papers

• D5RL: Diverse Datasets for Data-Driven Deep Reinforcement Learning [99.33607114541861]
  We propose a new benchmark for offline RL that focuses on realistic simulations of robotic manipulation and locomotion environments. Our proposed benchmark covers state-based and image-based domains, and supports both offline RL and online fine-tuning evaluation.
  arXiv  Detail & Related papers  (2024-08-15T22:27:00Z)
• The Challenges of Exploration for Offline Reinforcement Learning [8.484491887821473]
  We study the two processes of reinforcement learning: collecting informative experience and inferring optimal behaviour. The task-agnostic setting for data collection, where the task is not known a priori, is of particular interest. We use this decoupled framework to strengthen intuitions about exploration and the data prerequisites for effective offline RL.
  arXiv  Detail & Related papers  (2022-01-27T23:59:56Z)
• Representation Matters: Offline Pretraining for Sequential Decision Making [27.74988221252854]
  In this paper, we consider a slightly different approach to incorporating offline data into sequential decision-making. We find that the use of pretraining with unsupervised learning objectives can dramatically improve the performance of policy learning algorithms.
  arXiv  Detail & Related papers  (2021-02-11T02:38:12Z)
• Offline Reinforcement Learning from Images with Latent Space Models [60.69745540036375]
  offline reinforcement learning (RL) refers to the problem of learning policies from a static dataset of environment interactions. We build on recent advances in model-based algorithms for offline RL, and extend them to high-dimensional visual observation spaces. Our approach is both tractable in practice and corresponds to maximizing a lower bound of the ELBO in the unknown POMDP.
  arXiv  Detail & Related papers  (2020-12-21T18:28:17Z)
• OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning [107.6943868812716]
  In many practical applications, the situation is reversed: an agent may have access to large amounts of undirected offline experience data, while access to the online environment is severely limited. Our main insight is that, when presented with offline data composed of a variety of behaviors, an effective way to leverage this data is to extract a continuous space of recurring and temporally extended primitive behaviors. In addition to benefiting offline policy optimization, we show that performing offline primitive learning in this way can also be leveraged for improving few-shot imitation learning.
  arXiv  Detail & Related papers  (2020-10-26T14:31:08Z)
• RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning [108.9599280270704]
  We propose a benchmark called RL Unplugged to evaluate and compare offline RL methods. RL Unplugged includes data from a diverse range of domains including games and simulated motor control problems. We will release data for all our tasks and open-source all algorithms presented in this paper.
  arXiv  Detail & Related papers  (2020-06-24T17:14:51Z)
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning [119.49182500071288]
  We introduce benchmarks specifically designed for the offline setting, guided by key properties of datasets relevant to real-world applications of offline RL. By moving beyond simple benchmark tasks and data collected by partially-trained RL agents, we reveal important and unappreciated deficiencies of existing algorithms.
  arXiv  Detail & Related papers  (2020-04-15T17:18:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.