OLLIE: Imitation Learning from Offline Pretraining to Online Finetuning

• URL: http://arxiv.org/abs/2405.17477v3
• Date: Thu, 30 May 2024 17:11:46 GMT
• Title: OLLIE: Imitation Learning from Offline Pretraining to Online Finetuning
• Abstract summary: We study offline-to-online Imitation Learning (IL) that pretrains an imitation policy from static demonstration data, followed by fast finetuning. We propose a principled offline-to-online IL method, named $textttOLLIE$, that simultaneously learns a near-expert policy. Empirically, $textttOLLIE$ consistently and significantly outperforms the baseline methods in $textbf20$ challenging tasks.
• Score: 40.070791824270266
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper, we study offline-to-online Imitation Learning (IL) that pretrains an imitation policy from static demonstration data, followed by fast finetuning with minimal environmental interaction. We find the na\"ive combination of existing offline IL and online IL methods tends to behave poorly in this context, because the initial discriminator (often used in online IL) operates randomly and discordantly against the policy initialization, leading to misguided policy optimization and $\textit{unlearning}$ of pretraining knowledge. To overcome this challenge, we propose a principled offline-to-online IL method, named $\texttt{OLLIE}$, that simultaneously learns a near-expert policy initialization along with an $\textit{aligned discriminator initialization}$, which can be seamlessly integrated into online IL, achieving smooth and fast finetuning. Empirically, $\texttt{OLLIE}$ consistently and significantly outperforms the baseline methods in $\textbf{20}$ challenging tasks, from continuous control to vision-based domains, in terms of performance, demonstration efficiency, and convergence speed. This work may serve as a foundation for further exploration of pretraining and finetuning in the context of IL.

Related papers

• When Does Online Imitation Learning Help in LLM Post-Training? The Role of (Non-)Realizability Beyond Horizon [30.903287230763212]
  We show that the benefits of online interaction depend critically on whether the setting is realizable.<n>Under realizability, we empirically find that offline IL already matches expert performance.<n>We propose a structural characterization of misspecification relative to the reward, under which online IL achieves provably high performance.
  arXiv  Detail & Related papers  (2026-06-29T15:17:42Z)
• Action-Free Offline-to-Online RL via Discretised State Policies [8.696862667760778]
  We formalise the setting of action-free offline-to-online RL, where agents must learn from datasets consisting solely of $(s,r,s'$s)$s.<n>We propose learning state policies that recommend desirable next-state transitions rather than actions.
  arXiv  Detail & Related papers  (2026-01-31T09:53:04Z)
• Grounded in Reality: Learning and Deploying Proactive LLM from Offline Logs [72.08224879435762]
  textttLearn-to-Ask is a simulator-free framework for learning and deploying proactive dialogue agents.<n>Our approach culminates in the successful deployment of LLMs into a live, large-scale online AI service.
  arXiv  Detail & Related papers  (2025-10-29T12:08:07Z)
• Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RL [36.65926744075032]
  offline-to-online (O2O) reinforcement learning improves performance rapidly with limited online interactions. Recent studies often design fine-tuning strategies for a specific offline RL method and cannot perform general O2O learning from any offline method. We propose to handle these two mismatches simultaneously, which aims to achieve general O2O learning from any offline method to any online method.
  arXiv  Detail & Related papers  (2024-12-25T09:52:22Z)
• Bayesian Design Principles for Offline-to-Online Reinforcement Learning [50.97583504192167]
  offline-to-online fine-tuning is crucial for real-world applications where exploration can be costly or unsafe. In this paper, we tackle the dilemma of offline-to-online fine-tuning: if the agent remains pessimistic, it may fail to learn a better policy, while if it becomes optimistic directly, performance may suffer from a sudden drop. We show that Bayesian design principles are crucial in solving such a dilemma.
  arXiv  Detail & Related papers  (2024-05-31T16:31:07Z)
• Understanding the performance gap between online and offline alignment algorithms [63.137832242488926]
  We show that offline algorithms train policy to become good at pairwise classification, while online algorithms are good at generations. This hints at a unique interplay between discriminative and generative capabilities, which is greatly impacted by the sampling process. Our study sheds light on the pivotal role of on-policy sampling in AI alignment, and hints at certain fundamental challenges of offline alignment algorithms.
  arXiv  Detail & Related papers  (2024-05-14T09:12:30Z)
• Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimization [24.969834057981046]
  Previous approaches treat offline and online learning as separate procedures, resulting in redundant designs and limited performance. We propose Uni-o4, which utilizes an on-policy objective for both offline and online learning. We demonstrate that our method achieves state-of-the-art performance in both offline and offline-to-online fine-tuning learning.
  arXiv  Detail & Related papers  (2023-11-06T18:58:59Z)
• On Efficient Online Imitation Learning via Classification [17.416831207557603]
  We study classification-based online imitation learning (abbrev. $textbfCOIL$) and the fundamental feasibility to design oracle-efficient regret-minimization algorithms. Our work puts classification-based online imitation learning, an important IL setup, into a firmer foundation.
  arXiv  Detail & Related papers  (2022-09-26T17:34:36Z)
• Curriculum Offline Imitation Learning [72.1015201041391]
  offline reinforcement learning tasks require the agent to learn from a pre-collected dataset with no further interactions with the environment. We propose textitCurriculum Offline Learning (COIL), which utilizes an experience picking strategy for imitating from adaptive neighboring policies with a higher return. On continuous control benchmarks, we compare COIL against both imitation-based and RL-based methods, showing that it not only avoids just learning a mediocre behavior on mixed datasets but is also even competitive with state-of-the-art offline RL methods.
  arXiv  Detail & Related papers  (2021-11-03T08:02:48Z)
• Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coverage [27.122391441921664]
  This paper studies offline Imitation Learning (IL) where an agent learns to imitate an expert demonstrator without additional online environment interactions. Instead, the learner is presented with a static offline dataset of state-action-next state transition triples from a potentially less proficient behavior policy. We introduce Model-based IL from Offline data (MILO) to solve the offline IL problem efficiently both in theory and in practice.
  arXiv  Detail & Related papers  (2021-06-06T18:31:08Z)
• OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning [107.6943868812716]
  In many practical applications, the situation is reversed: an agent may have access to large amounts of undirected offline experience data, while access to the online environment is severely limited. Our main insight is that, when presented with offline data composed of a variety of behaviors, an effective way to leverage this data is to extract a continuous space of recurring and temporally extended primitive behaviors. In addition to benefiting offline policy optimization, we show that performing offline primitive learning in this way can also be leveraged for improving few-shot imitation learning.
  arXiv  Detail & Related papers  (2020-10-26T14:31:08Z)
• Explaining Fast Improvement in Online Imitation Learning [44.03882968581369]
  Online imitation learning (IL) is a framework that leverages interactions with expert policies for efficient policy optimization. Online IL has demonstrated empirical successes in many applications. We show that adopting a policy class in online IL has two benefits: both the policy improvement speed increases and the performance bias decreases.
  arXiv  Detail & Related papers  (2020-07-06T04:37:03Z)
• Non-Stationary Off-Policy Optimization [50.41335279896062]
  We study the novel problem of off-policy optimization in piecewise-stationary contextual bandits. In the offline learning phase, we partition logged data into categorical latent states and learn a near-optimal sub-policy for each state. In the online deployment phase, we adaptively switch between the learned sub-policies based on their performance.
  arXiv  Detail & Related papers  (2020-06-15T09:16:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.