Behavior Cloning is Not All You Need: The Optimality of On-Policy Distillation for Noisy Expert Feedback

• URL: http://arxiv.org/abs/2606.30923v1
• Date: Mon, 29 Jun 2026 21:18:21 GMT
• Title: Behavior Cloning is Not All You Need: The Optimality of On-Policy Distillation for Noisy Expert Feedback
• Abstract summary: In theory offline IL can be horizon-free and optimal, while online methods such as on-policy distillation often outperform offline methods such as supervised fine-tuning.<n>We propose a noisy expert model to explain this gap, in which the only learner has access to a noisy version of the expert's policy.<n>We prove that online interaction with the noisy expert via a novel variant of OPD enables dependence on the horizon in general.
• Score: 12.925021362985989
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Imitation Learning is a natural framework for learning in sequential decision-making systems and has emerged as the dominant paradigm through which we understand language model training. A central puzzle is that, while in theory offline IL can be horizon-free and optimal, in practice online methods such as on-policy distillation often outperform offline methods such as supervised fine-tuning. We propose a noisy expert model to explain this gap, in which the learner only has access to a noisy version of the expert's policy, but wishes to compete against the reward achieved by a clean expert, motivated by the fact that in many applications, e.g. training language models to perform long chains of thought, the expert is often imperfect. In this setting, we show a sharp separation between offline and online IL. Offline learning from noisy trajectories is fundamentally hard: to compete with the clean expert, the sample complexity must grow exponentially, in contradistinction to the clean expert setting where no explicit horizon dependence exists. In contrast, we prove that online interaction with the noisy expert via a novel variant of OPD enables polynomial dependence on the horizon in general. We further show that, under a natural condition on the expert noise distribution, which we show to be necessary for any horizon-free sample complexity, one can obtain such a guarantee, although our proposed algorithm sacrifices statistical efficiency in its dependence on the size of the policy class. Our analysis leads to an alternative loss function that is commonly considered empirically for LM training. We further provide algorithms and lower bounds, and extend our results to the more realistic setting of unknown corruption when the clean expert is deterministic, thereby providing a theoretical foundation for why OPD can outperform SFT when training language models from imperfect teachers.

Related papers

• When Does On-Policy Interaction Help? Representational Tradeoffs in Value-Based Imitation Learning [70.72643569128316]
  Imitation learning (IL) training an agent to replicate expert behavior from demonstrations.<n>We introduce OVI, an interactive on-policy IL algorithm that is statistically efficient whenever the learner can represent the expert's value function.<n>OVI outperforms offline policy-based (BC), interactive policy-based (DAgger), and offline value-based IL methods, with the largest gains when the learner network is substantially less expressive than the expert's.
  arXiv  Detail & Related papers  (2026-07-31T16:52:47Z)
• To Distill or Decide? Understanding the Algorithmic Trade-off in Partially Observable Reinforcement Learning [21.212850813246405]
  Partial observability is a notorious challenge in reinforcement learning (RL)<n>Recent empirical successes have used privileged expert distillation to learn and imitate the optimal latent, Markovian policy.<n>While expert distillation is more computationally efficient than RL without latent state information, it also has well-documented failure modes.
  arXiv  Detail & Related papers  (2025-10-03T17:45:47Z)
• Efficient Imitation under Misspecification [17.706710359787056]
  We consider the problem of imitation learning under misspecification.<n>We propose inverse reinforcement learning algorithms that merely perform a computationally efficient local search procedure with strong guarantees in the realizable setting.<n>We prove that in the misspecified setting, it is beneficial to broaden the set of states on which local search is performed to include those reachable by good policies the learner can actually play.
  arXiv  Detail & Related papers  (2025-03-17T13:35:55Z)
• The Pitfalls of Imitation Learning when Actions are Continuous [33.44344966171865]
  We study the problem of imitating an expert demonstrator in a continuous state-and-action control system.<n>We show that, even if the dynamics satisfy a control-theoretic property called exponential stability, any smooth, deterministic imitator policy necessarily suffers error.
  arXiv  Detail & Related papers  (2025-03-12T18:11:37Z)
• Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learning [26.53136644321385]
  Imitation learning (IL) aims to mimic the behavior of an expert in a sequential decision making task by learning from demonstrations.<n>We show that it is possible to achieve horizon-independent sample complexity in offline IL whenever the range of the cumulative payoffs is controlled.<n>Specializing in deterministic, stationary policies, we show that the gap between offline and online IL is smaller than previously thought.
  arXiv  Detail & Related papers  (2024-07-20T23:31:56Z)
• RLIF: Interactive Imitation Learning as Reinforcement Learning [56.997263135104504]
  We show how off-policy reinforcement learning can enable improved performance under assumptions that are similar but potentially even more practical than those of interactive imitation learning. Our proposed method uses reinforcement learning with user intervention signals themselves as rewards. This relaxes the assumption that intervening experts in interactive imitation learning should be near-optimal and enables the algorithm to learn behaviors that improve over the potential suboptimal human expert.
  arXiv  Detail & Related papers  (2023-11-21T21:05:21Z)
• Provable Guarantees for Generative Behavior Cloning: Bridging Low-Level Stability and High-Level Behavior [51.60683890503293]
  We propose a theoretical framework for studying behavior cloning of complex expert demonstrations using generative modeling. We show that pure supervised cloning can generate trajectories matching the per-time step distribution of arbitrary expert trajectories.
  arXiv  Detail & Related papers  (2023-07-27T04:27:26Z)
• When Demonstrations Meet Generative World Models: A Maximum Likelihood Framework for Offline Inverse Reinforcement Learning [62.00672284480755]
  This paper aims to recover the structure of rewards and environment dynamics that underlie observed actions in a fixed, finite set of demonstrations from an expert agent. Accurate models of expertise in executing a task has applications in safety-sensitive applications such as clinical decision making and autonomous driving.
  arXiv  Detail & Related papers  (2023-02-15T04:14:20Z)
• Efficient Performance Bounds for Primal-Dual Reinforcement Learning from Demonstrations [1.0609815608017066]
  We consider large-scale Markov decision processes with an unknown cost function and address the problem of learning a policy from a finite set of expert demonstrations. Existing inverse reinforcement learning methods come with strong theoretical guarantees, but are computationally expensive. We introduce a novel bilinear saddle-point framework using Lagrangian duality to bridge the gap between theory and practice.
  arXiv  Detail & Related papers  (2021-12-28T05:47:24Z)
• A Free Lunch from the Noise: Provable and Practical Exploration for Representation Learning [55.048010996144036]
  We show that under some noise assumption, we can obtain the linear spectral feature of its corresponding Markov transition operator in closed-form for free. We propose Spectral Dynamics Embedding (SPEDE), which breaks the trade-off and completes optimistic exploration for representation learning by exploiting the structure of the noise.
  arXiv  Detail & Related papers  (2021-11-22T19:24:57Z)
• Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximation [74.3002974673248]
  We consider the offline reinforcement learning problem, where the aim is to learn a decision making policy from logged data. offline RL is becoming increasingly relevant in practice, because online data collection is well suited to safety-critical domains. Our results show that sample-efficient offline reinforcement learning requires either restrictive coverage conditions or representation conditions that go beyond complexity learning.
  arXiv  Detail & Related papers  (2021-11-21T23:22:37Z)
• Adversarial Robustness with Semi-Infinite Constrained Learning [177.42714838799924]
  Deep learning to inputs perturbations has raised serious questions about its use in safety-critical domains. We propose a hybrid Langevin Monte Carlo training approach to mitigate this issue. We show that our approach can mitigate the trade-off between state-of-the-art performance and robust robustness.
  arXiv  Detail & Related papers  (2021-10-29T13:30:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.