NeoRL-2: Near Real-World Benchmarks for Offline Reinforcement Learning with Extended Realistic Scenarios

• URL: http://arxiv.org/abs/2503.19267v1
• Date: Tue, 25 Mar 2025 02:01:54 GMT
• Title: NeoRL-2: Near Real-World Benchmarks for Offline Reinforcement Learning with Extended Realistic Scenarios
• Authors: Songyi Gao, Zuolin Tu, Rong-Jun Qin, Yi-Hao Sun, Xiong-Hui Chen, Yang Yu,
• Abstract summary: offline reinforcement learning aims to learn from historical data without requiring (costly) access to the environment.<n>This benchmark consists of 7 datasets from 7 simulated tasks along with their corresponding evaluation simulators.<n>We hope NeoRL-2 will accelerate the development of reinforcement learning algorithms for real-world applications.
• Score: 8.93878940046993
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Offline reinforcement learning (RL) aims to learn from historical data without requiring (costly) access to the environment. To facilitate offline RL research, we previously introduced NeoRL, which highlighted that datasets from real-world tasks are often conservative and limited. With years of experience applying offline RL to various domains, we have identified additional real-world challenges. These include extremely conservative data distributions produced by deployed control systems, delayed action effects caused by high-latency transitions, external factors arising from the uncontrollable variance of transitions, and global safety constraints that are difficult to evaluate during the decision-making process. These challenges are underrepresented in previous benchmarks but frequently occur in real-world tasks. To address this, we constructed the extended Near Real-World Offline RL Benchmark (NeoRL-2), which consists of 7 datasets from 7 simulated tasks along with their corresponding evaluation simulators. Benchmarking results from state-of-the-art offline RL approaches demonstrate that current methods often struggle to outperform the data-collection behavior policy, highlighting the need for more effective methods. We hope NeoRL-2 will accelerate the development of reinforcement learning algorithms for real-world applications. The benchmark project page is available at https://github.com/polixir/NeoRL2.

Related papers

• D5RL: Diverse Datasets for Data-Driven Deep Reinforcement Learning [99.33607114541861]
  We propose a new benchmark for offline RL that focuses on realistic simulations of robotic manipulation and locomotion environments. Our proposed benchmark covers state-based and image-based domains, and supports both offline RL and online fine-tuning evaluation.
  arXiv  Detail & Related papers  (2024-08-15T22:27:00Z)
• Is Value Learning Really the Main Bottleneck in Offline RL? [70.54708989409409]
  We show that the choice of a policy extraction algorithm significantly affects the performance and scalability of offline RL. We propose two simple test-time policy improvement methods and show that these methods lead to better performance.
  arXiv  Detail & Related papers  (2024-06-13T17:07:49Z)
• Look Beneath the Surface: Exploiting Fundamental Symmetry for Sample-Efficient Offline RL [29.885978495034703]
  offline reinforcement learning (RL) offers an appealing approach to real-world tasks by learning policies from pre-collected datasets. However, the performance of existing offline RL algorithms heavily depends on the scale and state-action space coverage of datasets. We provide a new insight that leveraging the fundamental symmetry of system dynamics can substantially enhance offline RL performance under small datasets.
  arXiv  Detail & Related papers  (2023-06-07T07:51:05Z)
• Near Real-World Benchmarks for Offline Reinforcement Learning [26.642722521820467]
  We present a suite of near real-world benchmarks, NewRL. NewRL contains datasets from various domains with controlled sizes and extra test datasets for the purpose of policy validation. We argue that the performance of a policy should also be compared with the deterministic version of the behavior policy, instead of the dataset reward.
  arXiv  Detail & Related papers  (2021-02-01T09:19:10Z)
• Offline Reinforcement Learning from Images with Latent Space Models [60.69745540036375]
  offline reinforcement learning (RL) refers to the problem of learning policies from a static dataset of environment interactions. We build on recent advances in model-based algorithms for offline RL, and extend them to high-dimensional visual observation spaces. Our approach is both tractable in practice and corresponds to maximizing a lower bound of the ELBO in the unknown POMDP.
  arXiv  Detail & Related papers  (2020-12-21T18:28:17Z)
• Critic Regularized Regression [70.8487887738354]
  We propose a novel offline RL algorithm to learn policies from data using a form of critic-regularized regression (CRR) We find that CRR performs surprisingly well and scales to tasks with high-dimensional state and action spaces.
  arXiv  Detail & Related papers  (2020-06-26T17:50:26Z)
• RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning [108.9599280270704]
  We propose a benchmark called RL Unplugged to evaluate and compare offline RL methods. RL Unplugged includes data from a diverse range of domains including games and simulated motor control problems. We will release data for all our tasks and open-source all algorithms presented in this paper.
  arXiv  Detail & Related papers  (2020-06-24T17:14:51Z)
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning [119.49182500071288]
  We introduce benchmarks specifically designed for the offline setting, guided by key properties of datasets relevant to real-world applications of offline RL. By moving beyond simple benchmark tasks and data collected by partially-trained RL agents, we reveal important and unappreciated deficiencies of existing algorithms.
  arXiv  Detail & Related papers  (2020-04-15T17:18:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.