Robust Policy Expansion for Offline-to-Online RL under Diverse Data Corruption

• URL: http://arxiv.org/abs/2509.24748v2
• Date: Thu, 16 Oct 2025 10:59:56 GMT
• Title: Robust Policy Expansion for Offline-to-Online RL under Diverse Data Corruption
• Abstract summary: Offline-to-Online Reinforcement Learning (O2O RL) has emerged as a promising paradigm for real-world RL deployment.<n>Data corruption induces heavy-tailed behavior in the policy, thereby substantially degrading the efficiency of online exploration.<n>We propose a novel, simple yet effective method termed $textbfRPEX$: $textbfR$obust $textbfP$olicy $textbfEX$pansion.
• Score: 26.419116283028078
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Pretraining a policy on offline data followed by fine-tuning through online interactions, known as Offline-to-Online Reinforcement Learning (O2O RL), has emerged as a promising paradigm for real-world RL deployment. However, both offline datasets and online interactions in practical environments are often noisy or even maliciously corrupted, severely degrading the performance of O2O RL. Existing works primarily focus on mitigating the conservatism of offline policies via online exploration, while the robustness of O2O RL under data corruption, including states, actions, rewards, and dynamics, is still unexplored. In this work, we observe that data corruption induces heavy-tailed behavior in the policy, thereby substantially degrading the efficiency of online exploration. To address this issue, we incorporate Inverse Probability Weighted (IPW) into the online exploration policy to alleviate heavy-tailedness, and propose a novel, simple yet effective method termed $\textbf{RPEX}$: $\textbf{R}$obust $\textbf{P}$olicy $\textbf{EX}$pansion. Extensive experimental results on D4RL datasets demonstrate that RPEX achieves SOTA O2O performance across a wide range of data corruption scenarios. Code is available at $\href{https://github.com/felix-thu/RPEX}{https://github.com/felix-thu/RPEX}$.

Related papers

• Active Offline-to-Online Reinforcement Learning [17.49671322104596]
  We study active policy selection for fine-tuning under a limited interaction budget in offline reinforcement learning (O2O-RL) settings.<n>We propose an approach that balances this trade-off by actively selecting policies for fine-tuning based on upper-confidence bounds on their future performance.
  arXiv  Detail & Related papers  (2026-07-13T15:46:11Z)
• MOORL: A Framework for Integrating Offline-Online Reinforcement Learning [6.7265073544042995]
  We propose Meta Offline-Online Reinforcement Learning (MOORL), a hybrid framework that unifies offline and online learning.<n>Our theoretical analysis demonstrates that the hybrid approach enhances exploration by effectively combining the complementary strengths of offline and online data.<n>With minimal computational overhead, MOORL achieves strong performance, underscoring its potential for practical applications in real-world scenarios.
  arXiv  Detail & Related papers  (2025-06-11T10:12:50Z)
• Active Advantage-Aligned Online Reinforcement Learning with Offline Data [56.98480620108727]
  We introduce A3RL, which incorporates a novel confidence aware Active Advantage Aligned sampling strategy.<n>We demonstrate that our method outperforms competing online RL techniques that leverage offline data.
  arXiv  Detail & Related papers  (2025-02-11T20:31:59Z)
• Marvel: Accelerating Safe Online Reinforcement Learning with Finetuned Offline Policy [12.589890916332196]
  offline-to-online (O2O) RL can be leveraged to facilitate faster and safer online policy learning.<n>We introduce textbfMarvel, a novel framework for O2O safe RL, comprising two key components that work in concert.<n>Our work has the great potential to advance the field towards more efficient and practical safe RL solutions.
  arXiv  Detail & Related papers  (2024-12-05T18:51:18Z)
• Unsupervised-to-Online Reinforcement Learning [59.910638327123394]
  Unsupervised-to-online RL (U2O RL) replaces domain-specific supervised offline RL with unsupervised offline RL. U2O RL not only enables reusing a single pre-trained model for multiple downstream tasks, but also learns better representations. We empirically demonstrate that U2O RL achieves strong performance that matches or even outperforms previous offline-to-online RL approaches.
  arXiv  Detail & Related papers  (2024-08-27T05:23:45Z)
• D5RL: Diverse Datasets for Data-Driven Deep Reinforcement Learning [99.33607114541861]
  We propose a new benchmark for offline RL that focuses on realistic simulations of robotic manipulation and locomotion environments. Our proposed benchmark covers state-based and image-based domains, and supports both offline RL and online fine-tuning evaluation.
  arXiv  Detail & Related papers  (2024-08-15T22:27:00Z)
• Offline Retraining for Online RL: Decoupled Policy Learning to Mitigate Exploration Bias [96.14064037614942]
  offline retraining, a policy extraction step at the end of online fine-tuning, is proposed. An optimistic (exploration) policy is used to interact with the environment, and a separate pessimistic (exploitation) policy is trained on all the observed data for evaluation.
  arXiv  Detail & Related papers  (2023-10-12T17:50:09Z)
• Planning to Go Out-of-Distribution in Offline-to-Online Reinforcement Learning [9.341618348621662]
  We aim to find the best-performing policy within a limited budget of online interactions. We first study the major online RL exploration methods based on intrinsic rewards and UCB. We then introduce an algorithm for planning to go out-of-distribution that avoids these issues.
  arXiv  Detail & Related papers  (2023-10-09T13:47:05Z)
• Reward-agnostic Fine-tuning: Provable Statistical Benefits of Hybrid Reinforcement Learning [66.43003402281659]
  A central question boils down to how to efficiently utilize online data collection to strengthen and complement the offline dataset. We design a three-stage hybrid RL algorithm that beats the best of both worlds -- pure offline RL and pure online RL. The proposed algorithm does not require any reward information during data collection.
  arXiv  Detail & Related papers  (2023-05-17T15:17:23Z)
• On the Role of Discount Factor in Offline Reinforcement Learning [25.647624787936028]
  The discount factor, $gamma$, plays a vital role in improving online RL sample efficiency and estimation accuracy. This paper examines two distinct effects of $gamma$ in offline RL with theoretical analysis. The results show that the discount factor plays an essential role in the performance of offline RL algorithms.
  arXiv  Detail & Related papers  (2022-06-07T15:22:42Z)
• OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning [107.6943868812716]
  In many practical applications, the situation is reversed: an agent may have access to large amounts of undirected offline experience data, while access to the online environment is severely limited. Our main insight is that, when presented with offline data composed of a variety of behaviors, an effective way to leverage this data is to extract a continuous space of recurring and temporally extended primitive behaviors. In addition to benefiting offline policy optimization, we show that performing offline primitive learning in this way can also be leveraged for improving few-shot imitation learning.
  arXiv  Detail & Related papers  (2020-10-26T14:31:08Z)
• Critic Regularized Regression [70.8487887738354]
  We propose a novel offline RL algorithm to learn policies from data using a form of critic-regularized regression (CRR) We find that CRR performs surprisingly well and scales to tasks with high-dimensional state and action spaces.
  arXiv  Detail & Related papers  (2020-06-26T17:50:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.