Explaining Fast Improvement in Online Imitation Learning

• URL: http://arxiv.org/abs/2007.02520v3
• Date: Mon, 22 Feb 2021 01:26:47 GMT
• Title: Explaining Fast Improvement in Online Imitation Learning
• Abstract summary: Online imitation learning (IL) is a framework that leverages interactions with expert policies for efficient policy optimization. Online IL has demonstrated empirical successes in many applications. We show that adopting a policy class in online IL has two benefits: both the policy improvement speed increases and the performance bias decreases.
• Score: 44.03882968581369
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Online imitation learning (IL) is an algorithmic framework that leverages interactions with expert policies for efficient policy optimization. Here policies are optimized by performing online learning on a sequence of loss functions that encourage the learner to mimic expert actions, and if the online learning has no regret, the agent can provably learn an expert-like policy. Online IL has demonstrated empirical successes in many applications and interestingly, its policy improvement speed observed in practice is usually much faster than existing theory suggests. In this work, we provide an explanation of this phenomenon. Let $\xi$ denote the policy class bias and assume the online IL loss functions are convex, smooth, and non-negative. We prove that, after $N$ rounds of online IL with stochastic feedback, the policy improves in $\tilde{O}(1/N + \sqrt{\xi/N})$ in both expectation and high probability. In other words, we show that adopting a sufficiently expressive policy class in online IL has two benefits: both the policy improvement speed increases and the performance bias decreases.

Related papers

• When Does On-Policy Interaction Help? Representational Tradeoffs in Value-Based Imitation Learning [70.72643569128316]
  Imitation learning (IL) training an agent to replicate expert behavior from demonstrations.<n>We introduce OVI, an interactive on-policy IL algorithm that is statistically efficient whenever the learner can represent the expert's value function.<n>OVI outperforms offline policy-based (BC), interactive policy-based (DAgger), and offline value-based IL methods, with the largest gains when the learner network is substantially less expressive than the expert's.
  arXiv  Detail & Related papers  (2026-07-31T16:52:47Z)
• When Does Online Imitation Learning Help in LLM Post-Training? The Role of (Non-)Realizability Beyond Horizon [30.903287230763212]
  We show that the benefits of online interaction depend critically on whether the setting is realizable.<n>Under realizability, we empirically find that offline IL already matches expert performance.<n>We propose a structural characterization of misspecification relative to the reward, under which online IL achieves provably high performance.
  arXiv  Detail & Related papers  (2026-06-29T15:17:42Z)
• Bayesian Design Principles for Offline-to-Online Reinforcement Learning [50.97583504192167]
  offline-to-online fine-tuning is crucial for real-world applications where exploration can be costly or unsafe. In this paper, we tackle the dilemma of offline-to-online fine-tuning: if the agent remains pessimistic, it may fail to learn a better policy, while if it becomes optimistic directly, performance may suffer from a sudden drop. We show that Bayesian design principles are crucial in solving such a dilemma.
  arXiv  Detail & Related papers  (2024-05-31T16:31:07Z)
• OLLIE: Imitation Learning from Offline Pretraining to Online Finetuning [40.070791824270266]
  We study offline-to-online Imitation Learning (IL) that pretrains an imitation policy from static demonstration data, followed by fast finetuning. We propose a principled offline-to-online IL method, named $textttOLLIE$, that simultaneously learns a near-expert policy. Empirically, $textttOLLIE$ consistently and significantly outperforms the baseline methods in $textbf20$ challenging tasks.
  arXiv  Detail & Related papers  (2024-05-24T04:57:25Z)
• Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective [55.36819597141271]
  Inverse Reinforcement Learning (IRL) -- the problem of learning reward functions from demonstrations of an emphexpert policy -- plays a critical role in developing intelligent systems. This paper provides the first line of efficient IRL in vanilla offline and online settings using samples and runtime. As an application, we show that the learned rewards can emphtransfer to another target MDP with suitable guarantees.
  arXiv  Detail & Related papers  (2023-11-29T00:09:01Z)
• Offline Retraining for Online RL: Decoupled Policy Learning to Mitigate Exploration Bias [96.14064037614942]
  offline retraining, a policy extraction step at the end of online fine-tuning, is proposed. An optimistic (exploration) policy is used to interact with the environment, and a separate pessimistic (exploitation) policy is trained on all the observed data for evaluation.
  arXiv  Detail & Related papers  (2023-10-12T17:50:09Z)
• Boosting Offline Reinforcement Learning with Action Preference Query [32.94932149345299]
  Training practical agents usually involve offline and online reinforcement learning (RL) to balance the policy's performance and interaction costs. Online fine-tuning has become a commonly used method to correct the erroneous estimates of out-of-distribution data learned in the offline training phase. In this work, we introduce an interaction-free training scheme dubbed Offline-with-Action-Preferences (OAP)
  arXiv  Detail & Related papers  (2023-06-06T02:29:40Z)
• Smooth Imitation Learning via Smooth Costs and Smooth Policies [21.673771194165276]
  Imitation learning (IL) is a popular approach in the continuous control setting. In IL from demonstrations, an important challenge is to obtain agent policies that are smooth with respect to the inputs. We introduce a novel metric to quantify the smoothness of the learned policies.
  arXiv  Detail & Related papers  (2021-11-03T17:12:47Z)
• Curriculum Offline Imitation Learning [72.1015201041391]
  offline reinforcement learning tasks require the agent to learn from a pre-collected dataset with no further interactions with the environment. We propose textitCurriculum Offline Learning (COIL), which utilizes an experience picking strategy for imitating from adaptive neighboring policies with a higher return. On continuous control benchmarks, we compare COIL against both imitation-based and RL-based methods, showing that it not only avoids just learning a mediocre behavior on mixed datasets but is also even competitive with state-of-the-art offline RL methods.
  arXiv  Detail & Related papers  (2021-11-03T08:02:48Z)
• Combining Online Learning and Offline Learning for Contextual Bandits with Deficient Support [53.11601029040302]
  Current offline-policy learning algorithms are mostly based on inverse propensity score (IPS) weighting. We propose a novel approach that uses a hybrid of offline learning with online exploration. Our approach determines an optimal policy with theoretical guarantees using the minimal number of online explorations.
  arXiv  Detail & Related papers  (2021-07-24T05:07:43Z)
• Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coverage [27.122391441921664]
  This paper studies offline Imitation Learning (IL) where an agent learns to imitate an expert demonstrator without additional online environment interactions. Instead, the learner is presented with a static offline dataset of state-action-next state transition triples from a potentially less proficient behavior policy. We introduce Model-based IL from Offline data (MILO) to solve the offline IL problem efficiently both in theory and in practice.
  arXiv  Detail & Related papers  (2021-06-06T18:31:08Z)
• Off-Policy Adversarial Inverse Reinforcement Learning [0.0]
  Adversarial Imitation Learning (AIL) is a class of algorithms in Reinforcement learning (RL) We propose an Off-Policy Adversarial Inverse Reinforcement Learning (Off-policy-AIRL) algorithm which is sample efficient as well as gives good imitation performance.
  arXiv  Detail & Related papers  (2020-05-03T16:51:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.