SAMG: Offline-to-Online Reinforcement Learning via State-Action-Conditional Offline Model Guidance

• URL: http://arxiv.org/abs/2410.18626v2
• Date: Fri, 21 Feb 2025 11:46:46 GMT
• Title: SAMG: Offline-to-Online Reinforcement Learning via State-Action-Conditional Offline Model Guidance
• Authors: Liyu Zhang, Haochi Wu, Xu Wan, Quan Kong, Ruilong Deng, Mingyang Sun,
• Abstract summary: offline-to-online (O2O) reinforcement learning pre-trains models on offline data and refines policies through online fine-tuning.<n>We introduce State-Action-Conditional Offline Model Guidance (SAMG)<n>It freezes the pre-trained offline critic to provide compact offline understanding for each state-action sample.<n>It outperforms state-of-the-art O2O RL algorithms on the D4RL benchmark.
• Score: 10.78460888734411
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Offline-to-online (O2O) reinforcement learning (RL) pre-trains models on offline data and refines policies through online fine-tuning. However, existing O2O RL algorithms typically require maintaining the tedious offline datasets to mitigate the effects of out-of-distribution (OOD) data, which significantly limits their efficiency in exploiting online samples. To address this deficiency, we introduce a new paradigm for O2O RL called State-Action-Conditional Offline \Model Guidance (SAMG). It freezes the pre-trained offline critic to provide compact offline understanding for each state-action sample, thus eliminating the need for retraining on offline data. The frozen offline critic is incorporated with the online target critic weighted by a state-action-adaptive coefficient. This coefficient aims to capture the offline degree of samples at the state-action level, and is updated adaptively during training. In practice, SAMG could be easily integrated with Q-function-based algorithms. Theoretical analysis shows good optimality and lower estimation error. Empirically, SAMG outperforms state-of-the-art O2O RL algorithms on the D4RL benchmark.

Related papers

• Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RL [36.65926744075032]
  offline-to-online (O2O) reinforcement learning improves performance rapidly with limited online interactions. Recent studies often design fine-tuning strategies for a specific offline RL method and cannot perform general O2O learning from any offline method. We propose to handle these two mismatches simultaneously, which aims to achieve general O2O learning from any offline method to any online method.
  arXiv  Detail & Related papers  (2024-12-25T09:52:22Z)
• Unsupervised-to-Online Reinforcement Learning [59.910638327123394]
  Unsupervised-to-online RL (U2O RL) replaces domain-specific supervised offline RL with unsupervised offline RL. U2O RL not only enables reusing a single pre-trained model for multiple downstream tasks, but also learns better representations. We empirically demonstrate that U2O RL achieves strong performance that matches or even outperforms previous offline-to-online RL approaches.
  arXiv  Detail & Related papers  (2024-08-27T05:23:45Z)
• A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning [54.48409201256968]
  offline-to-online Reinforcement Learning (O2O RL) aims to improve the performance of offline pretrained policy using only a few online samples. Most O2O methods focus on the balance between RL objective and pessimism, or the utilization of offline and online samples.
  arXiv  Detail & Related papers  (2023-12-12T19:24:35Z)
• Train Once, Get a Family: State-Adaptive Balances for Offline-to-Online Reinforcement Learning [71.02384943570372]
  Family Offline-to-Online RL (FamO2O) is a framework that empowers existing algorithms to determine state-adaptive improvement-constraint balances. FamO2O offers a statistically significant improvement over various existing methods, achieving state-of-the-art performance on the D4RL benchmark.
  arXiv  Detail & Related papers  (2023-10-27T08:30:54Z)
• CROP: Conservative Reward for Model-based Offline Policy Optimization [15.121328040092264]
  This paper proposes a novel model-based offline RL algorithm, Conservative Reward for model-based Offline Policy optimization (CROP) To achieve a conservative reward estimation, CROP simultaneously minimizes the estimation error and the reward of random actions. Notably, CROP establishes an innovative connection between offline and online RL, highlighting that offline RL problems can be tackled by adopting online RL techniques.
  arXiv  Detail & Related papers  (2023-10-26T08:45:23Z)
• Towards Robust Offline-to-Online Reinforcement Learning via Uncertainty and Smoothness [11.241036026084222]
  offline-to-online (O2O) RL provides a paradigm for improving an offline trained agent within limited online interactions. Most offline RL algorithms suffer from performance drops and fail to achieve stable policy improvement in O2O adaptation. We propose the Robust Offline-to-Online (RO2O) algorithm, designed to enhance offline policies through uncertainty and smoothness.
  arXiv  Detail & Related papers  (2023-09-29T04:42:50Z)
• A Simple Unified Uncertainty-Guided Framework for Offline-to-Online Reinforcement Learning [25.123237633748193]
  offline-to-online reinforcement learning can be challenging due to constrained exploratory behavior and state-action distribution shift. We propose a Simple Unified uNcertainty-Guided (SUNG) framework, which unifies the solution to both challenges with the tool of uncertainty. SUNG achieves state-of-the-art online finetuning performance when combined with different offline RL methods.
  arXiv  Detail & Related papers  (2023-06-13T05:22:26Z)
• ENOTO: Improving Offline-to-Online Reinforcement Learning with Q-Ensembles [52.34951901588738]
  We propose a novel framework called ENsemble-based Offline-To-Online (ENOTO) RL. By increasing the number of Q-networks, we seamlessly bridge offline pre-training and online fine-tuning without degrading performance. Experimental results demonstrate that ENOTO can substantially improve the training stability, learning efficiency, and final performance of existing offline RL methods.
  arXiv  Detail & Related papers  (2023-06-12T05:10:10Z)
• Reward-agnostic Fine-tuning: Provable Statistical Benefits of Hybrid Reinforcement Learning [66.43003402281659]
  A central question boils down to how to efficiently utilize online data collection to strengthen and complement the offline dataset. We design a three-stage hybrid RL algorithm that beats the best of both worlds -- pure offline RL and pure online RL. The proposed algorithm does not require any reward information during data collection.
  arXiv  Detail & Related papers  (2023-05-17T15:17:23Z)
• Adaptive Behavior Cloning Regularization for Stable Offline-to-Online Reinforcement Learning [80.25648265273155]
  Offline reinforcement learning, by learning from a fixed dataset, makes it possible to learn agent behaviors without interacting with the environment. During online fine-tuning, the performance of the pre-trained agent may collapse quickly due to the sudden distribution shift from offline to online data. We propose to adaptively weigh the behavior cloning loss during online fine-tuning based on the agent's performance and training stability. Experiments show that the proposed method yields state-of-the-art offline-to-online reinforcement learning performance on the popular D4RL benchmark.
  arXiv  Detail & Related papers  (2022-10-25T09:08:26Z)
• Double Check Your State Before Trusting It: Confidence-Aware Bidirectional Offline Model-Based Imagination [31.805991958408438]
  We propose to augment the offline dataset by using trained bidirectional dynamics models and rollout policies with double check. Our method, confidence-aware bidirectional offline model-based imagination, generates reliable samples and can be combined with any model-free offline RL method.
  arXiv  Detail & Related papers  (2022-06-16T08:00:44Z)
• DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement Learning [17.664027379555183]
  offline reinforcement learning algorithms promise to be applicable in settings where a fixed dataset is available and no new experience can be acquired. This paper formulates the offline dynamics adaptation by using (source) offline data collected from another dynamics to relax the requirement for the extensive (target) offline data. With only modest amounts of target offline data, our performance consistently outperforms the prior offline RL methods in both simulated and real-world tasks.
  arXiv  Detail & Related papers  (2022-03-13T14:30:55Z)
• MOPO: Model-based Offline Policy Optimization [183.6449600580806]
  offline reinforcement learning (RL) refers to the problem of learning policies entirely from a large batch of previously collected data. We show that an existing model-based RL algorithm already produces significant gains in the offline setting. We propose to modify the existing model-based RL methods by applying them with rewards artificially penalized by the uncertainty of the dynamics.
  arXiv  Detail & Related papers  (2020-05-27T08:46:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.