Rainbow-DemoRL: Combining Improvements in Demonstration-Augmented Reinforcement Learning

• URL: http://arxiv.org/abs/2603.27400v1
• Date: Sat, 28 Mar 2026 20:34:20 GMT
• Title: Rainbow-DemoRL: Combining Improvements in Demonstration-Augmented Reinforcement Learning
• Abstract summary: Several approaches have been proposed to improve the sample efficiency of online reinforcement learning (RL) by leveraging demonstrations collected offline.<n>We classify existing demonstration-augmented RL approaches into three categories and perform an empirical study of their strengths, weaknesses, and combinations.<n>Our analysis reveals that directly reusing offline data and initializing with behavior cloning consistently outperform more complex offline RL pretraining methods for improving online sample efficiency.
• Score: 5.7784578751617275
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Several approaches have been proposed to improve the sample efficiency of online reinforcement learning (RL) by leveraging demonstrations collected offline. The offline data can be used directly as transitions to optimize RL objectives, or offline policy and value functions can first be learned from the data and then used for online finetuning or to provide reference actions. While each of these strategies has shown compelling results, it is unclear which method has the most impact on sample efficiency, whether these approaches can be combined, and if there are cumulative benefits. We classify existing demonstration-augmented RL approaches into three categories and perform an extensive empirical study of their strengths, weaknesses, and combinations to isolate the contribution of each strategy and determine effective hybrid combinations for sample-efficient online RL. Our analysis reveals that directly reusing offline data and initializing with behavior cloning consistently outperform more complex offline RL pretraining methods for improving online sample efficiency.

Related papers

• An Empirical Study on the Effectiveness of Incorporating Offline RL As Online RL Subroutines [8.277534985461477]
  We take the novel perspective of incorporating offline RL algorithms as subroutines of tabula rasa online RL.<n>This is feasible because an online learning agent can repurpose its historical interactions as offline dataset.<n>We formalize this idea into a framework that accommodates several variants of offline RL incorporation.
  arXiv  Detail & Related papers  (2025-11-29T08:17:03Z)
• From Static to Dynamic: Enhancing Offline-to-Online Reinforcement Learning via Energy-Guided Diffusion Stratification [3.2883573376133555]
  StratDiff is a diffusion model to learn prior knowledge from the offline dataset.<n>It refines this knowledge through energy-based functions to improve policy imitation and generate offline-like actions during online fine-tuning.<n>Offline-like samples are updated using offline objectives, while online-like samples follow online learning strategies.
  arXiv  Detail & Related papers  (2025-11-05T19:48:46Z)
• Bridging Offline and Online Reinforcement Learning for LLMs [71.48552761763158]
  We investigate the effectiveness of reinforcement learning methods for finetuning large language models when transitioning from offline to semi-online.<n>Our experiments cover training on verifiable math as well as non-verifiable instruction following with a set of benchmark evaluations for both.
  arXiv  Detail & Related papers  (2025-06-26T17:25:49Z)
• Unsupervised Data Generation for Offline Reinforcement Learning: A Perspective from Model [57.20064815347607]
  offline reinforcement learning (RL) recently gains growing interests from RL researchers.<n>The performance of offline RL suffers from the out-of-distribution problem, which can be corrected by feedback in online RL.<n>In this paper, we first build a bridge over the batch data and the performance of offline RL algorithms theoretically.<n>We show that in task-agnostic settings, a series of policies trained by unsupervised RL can minimize the worst-case regret in the performance gap.
  arXiv  Detail & Related papers  (2025-06-24T14:08:36Z)
• MOORL: A Framework for Integrating Offline-Online Reinforcement Learning [6.7265073544042995]
  We propose Meta Offline-Online Reinforcement Learning (MOORL), a hybrid framework that unifies offline and online learning.<n>Our theoretical analysis demonstrates that the hybrid approach enhances exploration by effectively combining the complementary strengths of offline and online data.<n>With minimal computational overhead, MOORL achieves strong performance, underscoring its potential for practical applications in real-world scenarios.
  arXiv  Detail & Related papers  (2025-06-11T10:12:50Z)
• What Matters for Batch Online Reinforcement Learning in Robotics? [65.06558240091758]
  The ability to learn from large batches of autonomously collected data for policy improvement holds the promise of enabling truly scalable robot learning.<n>Previous works have applied imitation learning and filtered imitation learning methods to the batch online RL problem.<n>We analyze how these axes affect performance and scaling with the amount of autonomous data.
  arXiv  Detail & Related papers  (2025-05-12T21:24:22Z)
• Active Advantage-Aligned Online Reinforcement Learning with Offline Data [56.98480620108727]
  We introduce A3RL, which incorporates a novel confidence aware Active Advantage Aligned sampling strategy.<n>We demonstrate that our method outperforms competing online RL techniques that leverage offline data.
  arXiv  Detail & Related papers  (2025-02-11T20:31:59Z)
• Reward-agnostic Fine-tuning: Provable Statistical Benefits of Hybrid Reinforcement Learning [66.43003402281659]
  A central question boils down to how to efficiently utilize online data collection to strengthen and complement the offline dataset. We design a three-stage hybrid RL algorithm that beats the best of both worlds -- pure offline RL and pure online RL. The proposed algorithm does not require any reward information during data collection.
  arXiv  Detail & Related papers  (2023-05-17T15:17:23Z)
• Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL [82.93243616342275]
  We introduce Offline Model-based RL with Adaptive Behavioral Priors (MABE) MABE is based on the finding that dynamics models, which support within-domain generalization, and behavioral priors, which support cross-domain generalization, are complementary. In experiments that require cross-domain generalization, we find that MABE outperforms prior methods.
  arXiv  Detail & Related papers  (2021-06-16T20:48:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.