Importance of Empirical Sample Complexity Analysis for Offline Reinforcement Learning

• URL: http://arxiv.org/abs/2112.15578v1
• Date: Fri, 31 Dec 2021 18:05:33 GMT
• Title: Importance of Empirical Sample Complexity Analysis for Offline Reinforcement Learning
• Authors: Samin Yeasar Arnob, Riashat Islam, Doina Precup
• Abstract summary: We ask the question of the dependency on the number of samples for learning from offline data. Our objective is to emphasize that studying sample complexity for offline RL is important, and is an indicator of the usefulness of existing offline algorithms.
• Score: 55.90351453865001
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We hypothesize that empirically studying the sample complexity of offline reinforcement learning (RL) is crucial for the practical applications of RL in the real world. Several recent works have demonstrated the ability to learn policies directly from offline data. In this work, we ask the question of the dependency on the number of samples for learning from offline data. Our objective is to emphasize that studying sample complexity for offline RL is important, and is an indicator of the usefulness of existing offline algorithms. We propose an evaluation approach for sample complexity analysis of offline RL.

Related papers

• Discovering Multiple Solutions from a Single Task in Offline Reinforcement Learning [51.00472376469131]
  We propose an algorithm that learns multiple solutions from a single task in offline reinforcement learning. Our experimental results show that the proposed algorithm learns multiple qualitatively and quantitatively distinctive solutions in offline RL.
  arXiv  Detail & Related papers  (2024-06-10T03:25:49Z)
• Bridging Distributionally Robust Learning and Offline RL: An Approach to Mitigate Distribution Shift and Partial Data Coverage [32.578787778183546]
  offline reinforcement learning (RL) algorithms learn optimal polices using historical (offline) data. One of the main challenges in offline RL is the distribution shift. We propose two offline RL algorithms using the distributionally robust learning (DRL) framework.
  arXiv  Detail & Related papers  (2023-10-27T19:19:30Z)
• Reward-agnostic Fine-tuning: Provable Statistical Benefits of Hybrid Reinforcement Learning [66.43003402281659]
  A central question boils down to how to efficiently utilize online data collection to strengthen and complement the offline dataset. We design a three-stage hybrid RL algorithm that beats the best of both worlds -- pure offline RL and pure online RL. The proposed algorithm does not require any reward information during data collection.
  arXiv  Detail & Related papers  (2023-05-17T15:17:23Z)
• Bridging the Gap Between Offline and Online Reinforcement Learning Evaluation Methodologies [6.303272140868826]
  Reinforcement learning (RL) has shown great promise with algorithms learning in environments with large state and action spaces. Current deep RL algorithms require a tremendous amount of environment interactions for learning. offline RL algorithms try to address this issue by bootstrapping the learning process from existing logged data.
  arXiv  Detail & Related papers  (2022-12-15T20:36:10Z)
• Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximation [74.3002974673248]
  We consider the offline reinforcement learning problem, where the aim is to learn a decision making policy from logged data. offline RL is becoming increasingly relevant in practice, because online data collection is well suited to safety-critical domains. Our results show that sample-efficient offline reinforcement learning requires either restrictive coverage conditions or representation conditions that go beyond complexity learning.
  arXiv  Detail & Related papers  (2021-11-21T23:22:37Z)
• Instabilities of Offline RL with Pre-Trained Neural Representation [127.89397629569808]
  In offline reinforcement learning (RL), we seek to utilize offline data to evaluate (or learn) policies in scenarios where the data are collected from a distribution that substantially differs from that of the target policy to be evaluated. Recent theoretical advances have shown that such sample-efficient offline RL is indeed possible provided certain strong representational conditions hold. This work studies these issues from an empirical perspective to gauge how stable offline RL methods are.
  arXiv  Detail & Related papers  (2021-03-08T18:06:44Z)
• Representation Matters: Offline Pretraining for Sequential Decision Making [27.74988221252854]
  In this paper, we consider a slightly different approach to incorporating offline data into sequential decision-making. We find that the use of pretraining with unsupervised learning objectives can dramatically improve the performance of policy learning algorithms.
  arXiv  Detail & Related papers  (2021-02-11T02:38:12Z)
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning [119.49182500071288]
  We introduce benchmarks specifically designed for the offline setting, guided by key properties of datasets relevant to real-world applications of offline RL. By moving beyond simple benchmark tasks and data collected by partially-trained RL agents, we reveal important and unappreciated deficiencies of existing algorithms.
  arXiv  Detail & Related papers  (2020-04-15T17:18:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.