When Does Online Imitation Learning Help in LLM Post-Training? The Role of (Non-)Realizability Beyond Horizon

• URL: http://arxiv.org/abs/2606.30445v1
• Date: Mon, 29 Jun 2026 15:17:42 GMT
• Title: When Does Online Imitation Learning Help in LLM Post-Training? The Role of (Non-)Realizability Beyond Horizon
• Abstract summary: We show that the benefits of online interaction depend critically on whether the setting is realizable.<n>Under realizability, we empirically find that offline IL already matches expert performance.<n>We propose a structural characterization of misspecification relative to the reward, under which online IL achieves provably high performance.
• Score: 30.903287230763212
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Online imitation learning (IL), particularly on-policy distillation, has emerged as a strong LLM post-training approach, often outperforming offline supervised fine-tuning (SFT). Yet a principled understanding of when and why online interaction helps remains unclear. In this work, we challenge the view that error accumulation is the main source of online IL's advantage, and instead show that the benefits of online interaction depend critically on whether the setting is realizable, i.e., whether the student policy class can represent the expert policy. Under realizability, we empirically find that offline IL already matches expert performance. In contrast, in non-realizable (misspecified) settings, we prove that offline IL encounters an information-theoretic bottleneck even when horizon $H=1$, and propose a structural characterization of misspecification relative to the reward, under which online IL provably achieves high performance despite a large distributional mismatch between the expert and student policies.

Related papers

• When Does On-Policy Interaction Help? Representational Tradeoffs in Value-Based Imitation Learning [70.72643569128316]
  Imitation learning (IL) training an agent to replicate expert behavior from demonstrations.<n>We introduce OVI, an interactive on-policy IL algorithm that is statistically efficient whenever the learner can represent the expert's value function.<n>OVI outperforms offline policy-based (BC), interactive policy-based (DAgger), and offline value-based IL methods, with the largest gains when the learner network is substantially less expressive than the expert's.
  arXiv  Detail & Related papers  (2026-07-31T16:52:47Z)
• Grounded in Reality: Learning and Deploying Proactive LLM from Offline Logs [72.08224879435762]
  textttLearn-to-Ask is a simulator-free framework for learning and deploying proactive dialogue agents.<n>Our approach culminates in the successful deployment of LLMs into a live, large-scale online AI service.
  arXiv  Detail & Related papers  (2025-10-29T12:08:07Z)
• Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learning [26.53136644321385]
  Imitation learning (IL) aims to mimic the behavior of an expert in a sequential decision making task by learning from demonstrations.<n>We show that it is possible to achieve horizon-independent sample complexity in offline IL whenever the range of the cumulative payoffs is controlled.<n>Specializing in deterministic, stationary policies, we show that the gap between offline and online IL is smaller than previously thought.
  arXiv  Detail & Related papers  (2024-07-20T23:31:56Z)
• OLLIE: Imitation Learning from Offline Pretraining to Online Finetuning [40.070791824270266]
  We study offline-to-online Imitation Learning (IL) that pretrains an imitation policy from static demonstration data, followed by fast finetuning. We propose a principled offline-to-online IL method, named $textttOLLIE$, that simultaneously learns a near-expert policy. Empirically, $textttOLLIE$ consistently and significantly outperforms the baseline methods in $textbf20$ challenging tasks.
  arXiv  Detail & Related papers  (2024-05-24T04:57:25Z)
• ENOTO: Improving Offline-to-Online Reinforcement Learning with Q-Ensembles [52.34951901588738]
  We propose a novel framework called ENsemble-based Offline-To-Online (ENOTO) RL. By increasing the number of Q-networks, we seamlessly bridge offline pre-training and online fine-tuning without degrading performance. Experimental results demonstrate that ENOTO can substantially improve the training stability, learning efficiency, and final performance of existing offline RL methods.
  arXiv  Detail & Related papers  (2023-06-12T05:10:10Z)
• MAHALO: Unifying Offline Reinforcement Learning and Imitation Learning from Observations [43.9636309593499]
  We study a new paradigm for sequential decision making, called offline policy learning from observations (PLfO) We present a generic approach to offline PLfO, called $textbfM$odality-agnostic $textbfA$dversarial $textbfH$ypothesis $textbfA$daptation for $textbfL$earning from $textbfO$bservations (MAHALO)
  arXiv  Detail & Related papers  (2023-03-30T05:27:46Z)
• Curriculum Offline Imitation Learning [72.1015201041391]
  offline reinforcement learning tasks require the agent to learn from a pre-collected dataset with no further interactions with the environment. We propose textitCurriculum Offline Learning (COIL), which utilizes an experience picking strategy for imitating from adaptive neighboring policies with a higher return. On continuous control benchmarks, we compare COIL against both imitation-based and RL-based methods, showing that it not only avoids just learning a mediocre behavior on mixed datasets but is also even competitive with state-of-the-art offline RL methods.
  arXiv  Detail & Related papers  (2021-11-03T08:02:48Z)
• Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coverage [27.122391441921664]
  This paper studies offline Imitation Learning (IL) where an agent learns to imitate an expert demonstrator without additional online environment interactions. Instead, the learner is presented with a static offline dataset of state-action-next state transition triples from a potentially less proficient behavior policy. We introduce Model-based IL from Offline data (MILO) to solve the offline IL problem efficiently both in theory and in practice.
  arXiv  Detail & Related papers  (2021-06-06T18:31:08Z)
• OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning [107.6943868812716]
  In many practical applications, the situation is reversed: an agent may have access to large amounts of undirected offline experience data, while access to the online environment is severely limited. Our main insight is that, when presented with offline data composed of a variety of behaviors, an effective way to leverage this data is to extract a continuous space of recurring and temporally extended primitive behaviors. In addition to benefiting offline policy optimization, we show that performing offline primitive learning in this way can also be leveraged for improving few-shot imitation learning.
  arXiv  Detail & Related papers  (2020-10-26T14:31:08Z)
• Explaining Fast Improvement in Online Imitation Learning [44.03882968581369]
  Online imitation learning (IL) is a framework that leverages interactions with expert policies for efficient policy optimization. Online IL has demonstrated empirical successes in many applications. We show that adopting a policy class in online IL has two benefits: both the policy improvement speed increases and the performance bias decreases.
  arXiv  Detail & Related papers  (2020-07-06T04:37:03Z)
• AliExpress Learning-To-Rank: Maximizing Online Model Performance without Going Online [60.887637616379926]
  This paper proposes an evaluator-generator framework for learning-to-rank. It consists of an evaluator that generalizes to evaluate recommendations involving the context, and a generator that maximizes the evaluator score by reinforcement learning. Our method achieves a significant improvement in terms of Conversion Rate (CR) over the industrial-level fine-tuned model in online A/B tests.
  arXiv  Detail & Related papers  (2020-03-25T10:27:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.