Critic-Free Pretraining for Efficient Online Reinforcement Learning Fine-Tuning

• URL: http://arxiv.org/abs/2608.10473v2
• Date: Thu, 13 Aug 2026 16:34:58 GMT
• Title: Critic-Free Pretraining for Efficient Online Reinforcement Learning Fine-Tuning
• Abstract summary: Critic-Free Pretraining is an efficient paradigm that completely abandons the approach of offline critic training.<n>It is compatible with various mainstream O2O algorithms and consistently matches or improves upon conventional O2O algorithms.
• Score: 8.965329606005263
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Offline-to-online (O2O) reinforcement learning aims to leverage policies pretrained on static datasets while improving them through online interaction. However, directly reusing an offline-trained critic can hinder online fine-tuning: as the policy and data distribution change rapidly, value estimates inherited from offline training may become misaligned with the online environment, leading to inaccurate policy improvement and inefficient exploration. To address this problem, we introduce Critic-Free Pretraining: an efficient paradigm that completely abandons the approach of offline critic training, allowing a freshly initialized critic to adapt without inheriting biased estimates. CFP is compatible with various mainstream O2O algorithms and consistently matches or improves upon conventional O2O algorithms across a diverse set of tasks, with particularly pronounced gains on several challenging tasks.

Related papers

• Active Offline-to-Online Reinforcement Learning [17.49671322104596]
  We study active policy selection for fine-tuning under a limited interaction budget in offline reinforcement learning (O2O-RL) settings.<n>We propose an approach that balances this trade-off by actively selecting policies for fine-tuning based on upper-confidence bounds on their future performance.
  arXiv  Detail & Related papers  (2026-07-13T15:46:11Z)
• ROAD: Adaptive Data Mixing for Offline-to-Online Reinforcement Learning via Bi-Level Optimization [19.46945184186446]
  We propose Reinforcement Learning with Optimized Adaptive Data-mixing (ROAD)<n>ROAD is a dynamic plug-and-play framework that automates the data replay process.<n>Our empirical results demonstrate that this approach consistently outperforms existing data replay methods across various datasets.
  arXiv  Detail & Related papers  (2026-05-14T07:35:58Z)
• SOPE: Stabilizing Off-Policy Evaluation for Online RL with Prior Data [49.370849653460716]
  We propose SOPE, an algorithm that uses an actor-aligned Off-Policy Policy Evaluation signal as an automated early-stopping mechanism.<n> evaluated on 25 continuous control tasks from the Minari benchmark suite.<n>SOPE improves baseline performance by up to 45.6% while reducing the required TFLOPs by up to 22x.
  arXiv  Detail & Related papers  (2026-05-07T08:32:09Z)
• Adaptive Policy Selection and Fine-Tuning under Interaction Budgets for Offline-to-Online Reinforcement Learning [22.717342103512493]
  In offline-to-online reinforcement learning (O2O-RL), policies are first safely trained offline using previously collected datasets.<n>We propose a novel adaptive approach for policy selection and fine-tuning under online interaction budgets in O2O-RL.
  arXiv  Detail & Related papers  (2026-05-06T16:51:44Z)
• Q-learning with Adjoint Matching [58.78551025170267]
  We propose Q-learning with Adjoint Matching (QAM), a novel TD-based reinforcement learning (RL) algorithm.<n>QAM sidesteps two challenges by leveraging adjoint matching, a recently proposed technique in generative modeling.<n>It consistently outperforms prior approaches on hard, sparse reward tasks in both offline and offline-to-online RL.
  arXiv  Detail & Related papers  (2026-01-20T18:45:34Z)
• Adversarial Fine-tuning in Offline-to-Online Reinforcement Learning for Robust Robot Control [12.961180148172199]
  This study introduces an offline-to-online framework that trains policies on clean data and then performs adversarial fine-tuning.<n>A performance-aware curriculum adjusts the perturbation probability during training via an exponential-moving-average signal.<n>Experiments on continuous-control locomotion tasks demonstrate that the proposed method consistently improves robustness over offline-only baselines.
  arXiv  Detail & Related papers  (2025-10-15T09:45:24Z)
• Online-BLS: An Accurate and Efficient Online Broad Learning System for Data Stream Classification [52.251569042852815]
  We introduce an online broad learning system framework with closed-form solutions for each online update.<n>We design an effective weight estimation algorithm and an efficient online updating strategy.<n>Our framework is naturally extended to data stream scenarios with concept drift and exceeds state-of-the-art baselines.
  arXiv  Detail & Related papers  (2025-01-28T13:21:59Z)
• Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RL [36.65926744075032]
  offline-to-online (O2O) reinforcement learning improves performance rapidly with limited online interactions.<n>Recent studies often design fine-tuning strategies for a specific offline RL method and cannot perform general O2O learning from any offline method.<n>We propose to handle these two mismatches simultaneously, which aims to achieve general O2O learning from any offline method to any online method.
  arXiv  Detail & Related papers  (2024-12-25T09:52:22Z)
• Small Dataset, Big Gains: Enhancing Reinforcement Learning by Offline Pre-Training with Model Based Augmentation [59.899714450049494]
  offline pre-training can produce sub-optimal policies and lead to degraded online reinforcement learning performance. We propose a model-based data augmentation strategy to maximize the benefits of offline reinforcement learning pre-training and reduce the scale of data needed to be effective.
  arXiv  Detail & Related papers  (2023-12-15T14:49:41Z)
• Statistically Efficient Variance Reduction with Double Policy Estimation for Off-Policy Evaluation in Sequence-Modeled Reinforcement Learning [53.97273491846883]
  We propose DPE: an RL algorithm that blends offline sequence modeling and offline reinforcement learning with Double Policy Estimation. We validate our method in multiple tasks of OpenAI Gym with D4RL benchmarks.
  arXiv  Detail & Related papers  (2023-08-28T20:46:07Z)
• Boosting Offline Reinforcement Learning with Action Preference Query [32.94932149345299]
  Training practical agents usually involve offline and online reinforcement learning (RL) to balance the policy's performance and interaction costs. Online fine-tuning has become a commonly used method to correct the erroneous estimates of out-of-distribution data learned in the offline training phase. In this work, we introduce an interaction-free training scheme dubbed Offline-with-Action-Preferences (OAP)
  arXiv  Detail & Related papers  (2023-06-06T02:29:40Z)
• Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions [30.050083797177706]
  offline reinforcement learning (RL) allows for the training of competent agents from offline datasets without any interaction with the environment. Online finetuning of such offline models can further improve performance. We show that it is possible to use standard online off-policy algorithms for faster improvement.
  arXiv  Detail & Related papers  (2023-03-30T14:08:31Z)
• Adaptive Behavior Cloning Regularization for Stable Offline-to-Online Reinforcement Learning [80.25648265273155]
  Offline reinforcement learning, by learning from a fixed dataset, makes it possible to learn agent behaviors without interacting with the environment. During online fine-tuning, the performance of the pre-trained agent may collapse quickly due to the sudden distribution shift from offline to online data. We propose to adaptively weigh the behavior cloning loss during online fine-tuning based on the agent's performance and training stability. Experiments show that the proposed method yields state-of-the-art offline-to-online reinforcement learning performance on the popular D4RL benchmark.
  arXiv  Detail & Related papers  (2022-10-25T09:08:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.