Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matters

• URL: http://arxiv.org/abs/2110.04156v1
• Date: Fri, 8 Oct 2021 14:25:21 GMT
• Title: Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matters
• Abstract summary: We propose an evaluation technique that provides a performance estimate for a best-found policy given a fixed online evaluation budget. We find that with a limited online evaluation budget, Behavioral Cloning constitutes a strong baseline over various expert levels and data regimes. We hope the proposed technique will make it into the toolsets of Offline-RL practitioners to help them arrive at informed conclusions when deploying RL in real-world systems.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Over the recent years, vast progress has been made in Offline Reinforcement Learning (Offline-RL) for various decision-making domains: from finance to robotics. However, comparing and reporting new Offline-RL algorithms has been noted as underdeveloped: (1) use of unlimited online evaluation budget for hyperparameter search (2) sidestepping offline policy selection (3) ad-hoc performance statistics reporting. In this work, we propose an evaluation technique addressing these issues, Expected Online Performance, that provides a performance estimate for a best-found policy given a fixed online evaluation budget. Using our approach, we can estimate the number of online evaluations required to surpass a given behavioral policy performance. Applying it to several Offline-RL baselines, we find that with a limited online evaluation budget, (1) Behavioral Cloning constitutes a strong baseline over various expert levels and data regimes, and (2) offline uniform policy selection is competitive with value-based approaches. We hope the proposed technique will make it into the toolsets of Offline-RL practitioners to help them arrive at informed conclusions when deploying RL in real-world systems.

Related papers

• Active Offline-to-Online Reinforcement Learning [17.49671322104596]
  We study active policy selection for fine-tuning under a limited interaction budget in offline reinforcement learning (O2O-RL) settings.<n>We propose an approach that balances this trade-off by actively selecting policies for fine-tuning based on upper-confidence bounds on their future performance.
  arXiv  Detail & Related papers  (2026-07-13T15:46:11Z)
• Adaptive Policy Selection and Fine-Tuning under Interaction Budgets for Offline-to-Online Reinforcement Learning [22.717342103512493]
  In offline-to-online reinforcement learning (O2O-RL), policies are first safely trained offline using previously collected datasets.<n>We propose a novel adaptive approach for policy selection and fine-tuning under online interaction budgets in O2O-RL.
  arXiv  Detail & Related papers  (2026-05-06T16:51:44Z)
• An Empirical Study on the Effectiveness of Incorporating Offline RL As Online RL Subroutines [8.277534985461477]
  We take the novel perspective of incorporating offline RL algorithms as subroutines of tabula rasa online RL.<n>This is feasible because an online learning agent can repurpose its historical interactions as offline dataset.<n>We formalize this idea into a framework that accommodates several variants of offline RL incorporation.
  arXiv  Detail & Related papers  (2025-11-29T08:17:03Z)
• Online Optimization for Offline Safe Reinforcement Learning [44.48700237186216]
  We study the problem of Offline Safe Reinforcement Learning (OSRL)<n>The goal is to learn a reward-maximizing policy from fixed data under a cumulative cost constraint.<n>We propose a novel OSRL approach that frames the problem as a minimax objective and solves it by combining offline RL with online optimization algorithms.
  arXiv  Detail & Related papers  (2025-10-24T21:12:47Z)
• Active Advantage-Aligned Online Reinforcement Learning with Offline Data [56.98480620108727]
  A3 RL is a novel method that actively selects data from combined online and offline sources to optimize policy improvement. We provide theoretical guarantee that validates the effectiveness of our active sampling strategy.
  arXiv  Detail & Related papers  (2025-02-11T20:31:59Z)
• Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RL [36.65926744075032]
  offline-to-online (O2O) reinforcement learning improves performance rapidly with limited online interactions. Recent studies often design fine-tuning strategies for a specific offline RL method and cannot perform general O2O learning from any offline method. We propose to handle these two mismatches simultaneously, which aims to achieve general O2O learning from any offline method to any online method.
  arXiv  Detail & Related papers  (2024-12-25T09:52:22Z)
• A Non-Monolithic Policy Approach of Offline-to-Online Reinforcement Learning [2.823645435281551]
  offline-to-online reinforcement learning (RL) uses both pre-trained offline policies and online policies trained for downstream tasks. In this study, we propose an innovative offline-to-online RL method that employs a non-monolithic exploration approach.
  arXiv  Detail & Related papers  (2024-10-31T08:49:37Z)
• Bayesian Design Principles for Offline-to-Online Reinforcement Learning [50.97583504192167]
  offline-to-online fine-tuning is crucial for real-world applications where exploration can be costly or unsafe. In this paper, we tackle the dilemma of offline-to-online fine-tuning: if the agent remains pessimistic, it may fail to learn a better policy, while if it becomes optimistic directly, performance may suffer from a sudden drop. We show that Bayesian design principles are crucial in solving such a dilemma.
  arXiv  Detail & Related papers  (2024-05-31T16:31:07Z)
• Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL [42.57662196581823]
  Off-policy reinforcement learning (RL) has achieved notable success in tackling many complex real-world tasks. Most existing off-policy RL algorithms fail to maximally exploit the information in the replay buffer. We present Offline-Boosted Actor-Critic (OBAC), a model-free online RL framework that elegantly identifies the outperforming offline policy.
  arXiv  Detail & Related papers  (2024-05-28T18:38:46Z)
• A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning [54.48409201256968]
  offline-to-online Reinforcement Learning (O2O RL) aims to improve the performance of offline pretrained policy using only a few online samples. Most O2O methods focus on the balance between RL objective and pessimism, or the utilization of offline and online samples.
  arXiv  Detail & Related papers  (2023-12-12T19:24:35Z)
• Offline Retraining for Online RL: Decoupled Policy Learning to Mitigate Exploration Bias [96.14064037614942]
  offline retraining, a policy extraction step at the end of online fine-tuning, is proposed. An optimistic (exploration) policy is used to interact with the environment, and a separate pessimistic (exploitation) policy is trained on all the observed data for evaluation.
  arXiv  Detail & Related papers  (2023-10-12T17:50:09Z)
• ENOTO: Improving Offline-to-Online Reinforcement Learning with Q-Ensembles [52.34951901588738]
  We propose a novel framework called ENsemble-based Offline-To-Online (ENOTO) RL. By increasing the number of Q-networks, we seamlessly bridge offline pre-training and online fine-tuning without degrading performance. Experimental results demonstrate that ENOTO can substantially improve the training stability, learning efficiency, and final performance of existing offline RL methods.
  arXiv  Detail & Related papers  (2023-06-12T05:10:10Z)
• Adaptive Policy Learning for Offline-to-Online Reinforcement Learning [27.80266207283246]
  We consider an offline-to-online setting where the agent is first learned from the offline dataset and then trained online. We propose a framework called Adaptive Policy Learning for effectively taking advantage of offline and online data.
  arXiv  Detail & Related papers  (2023-03-14T08:13:21Z)
• Pessimistic Model Selection for Offline Deep Reinforcement Learning [56.282483586473816]
  Deep Reinforcement Learning (DRL) has demonstrated great potentials in solving sequential decision making problems in many applications. One main barrier is the over-fitting issue that leads to poor generalizability of the policy learned by DRL. We propose a pessimistic model selection (PMS) approach for offline DRL with a theoretical guarantee.
  arXiv  Detail & Related papers  (2021-11-29T06:29:49Z)
• Benchmarks for Deep Off-Policy Evaluation [152.28569758144022]
  We present a collection of policies that can be used for benchmarking off-policy evaluation. The goal of our benchmark is to provide a standardized measure of progress that is motivated from a set of principles. We provide open-source access to our data and code to foster future research in this area.
  arXiv  Detail & Related papers  (2021-03-30T18:09:33Z)
• Representation Matters: Offline Pretraining for Sequential Decision Making [27.74988221252854]
  In this paper, we consider a slightly different approach to incorporating offline data into sequential decision-making. We find that the use of pretraining with unsupervised learning objectives can dramatically improve the performance of policy learning algorithms.
  arXiv  Detail & Related papers  (2021-02-11T02:38:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.