Multi-Turn On-Policy Distillation with Prefix Replay

• URL: http://arxiv.org/abs/2607.04763v1
• Date: Mon, 06 Jul 2026 07:56:53 GMT
• Title: Multi-Turn On-Policy Distillation with Prefix Replay
• Abstract summary: We propose Replayed-Prefix On-Policy Distillation (ReOPD) for agentic tasks.<n>ReOPD reuses pre-collected teacher trajectories as replayed prefixes.<n>It preserves or improves OPD-level accuracy, uses zero tool calls during student training, and is at least 4$times$ faster per training step than OPD.
• Score: 73.10000453999088
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We study on-policy distillation (OPD) for agentic tasks, where an LLM agent interacts with an environment over multiple turns and a student imitates a teacher over these multi-turn interaction histories. Fully online OPD is costly because each update requires fresh student rollouts through the environment and teacher queries at visited histories. We propose Replayed-Prefix On-Policy Distillation (ReOPD), an off-environment alternative that reuses pre-collected teacher trajectories as replayed prefixes: the student acts at selected steps, while the teacher provides dense per-step supervision without executing new environment interactions. We show that multi-turn OPD introduces a prefix trap: making histories more student-on-policy improves relevance to the student, but can query the teacher on histories where its target is unreliable. This creates a two-sided distribution shift between student occupancy and teacher reliability. ReOPD addresses this by treating multi-turn OPD as a reliability-aware prefix distribution design and implements it with a simple step-decaying sampling schedule that emphasizes early, lower-shift prefixes. Across mathematical reasoning with Python and search environments over multiple teacher and student model scales, ReOPD preserves or improves OPD-level accuracy, uses zero tool calls during student training, and is at least 4$\times$ faster per training step than OPD. ReOPD therefore turns expensive agent-environment interaction into a reusable offline resource, enabling scalable distillation across tools, tasks, and environments.

Related papers

• ReOrder-OPD:Reliability-Aware Prompt Ordering for On-Policy Distillation [25.69241346799495]
  We define prompt-level teacher continuation reliability $R$ as the teacher's probability of reaching a correct answer from a student prefix.<n>We show that high-$R$ prompts yield larger OPD gains and that descending-$R$ training outperforms random and ascending orders.
  arXiv  Detail & Related papers  (2026-08-11T13:27:56Z)
• DASH-OPD: Discrepancy-Aware Switching with Hysteresis for On-Policy Distillation [9.897015521661915]
  On-policy distillation (OPD) trains student models on their own rollouts to reduce exposure bias.<n>Existing curriculum learning methods regulate how much teacher support is used according to training progress, but cannot determine when it is needed.<n>We propose DASH-OPD, a new agentic OPD method that can switch executors adaptively and bidirectionally.
  arXiv  Detail & Related papers  (2026-07-31T06:57:17Z)
• dOPSD: On-Policy Self-Distillation for Diffusion Language Models [52.60302464420127]
  Diffusion large language models generate text by iteratively denoising a masked sequence.<n>Supervised fine-tuning is off-policy and suffers from exposure bias, while reinforcement learning gives only sparse, sequence-level rewards.<n>On-policy self-distillation (OPSD) offers a promising alternative, using one model as both student and teacher.
  arXiv  Detail & Related papers  (2026-07-05T17:47:29Z)
• Prefix-Guided On-Policy Distillation: Mining Golden Trajectories from Rollouts [48.550535291129584]
  Prefix-Guided On-Policy Distillation (PG-OPD) is a simple rollout-allocation framework that uses fixed-length prefixes to estimate trajectory value before expensive long-horizon generation.<n>Across diverse teacher-student combinations on AMC, AIME, and HMMT benchmarks, PG-OPD improves average accuracy by up to 4.80 points while reducing training time by up to 2.46x.
  arXiv  Detail & Related papers  (2026-06-20T11:18:34Z)
• SAGE-OPD: Selective Agent-Guided Intervention for Multi-Turn On-Policy Distillation [25.25989941933095]
  We propose SAGE-OPD, a verifier-free selective intervention framework specifically designed for multi-turn OPD.<n>Instead of applying teacher supervision uniformly across all turns, SAGE-OPD first observes environment feedback and uses teacher judgment to decide whether each student response should be skipped or intervened on.<n>Experiments on agent tasks show that SAGE-OPD consistently improves over baselines, achieving up to a 13.3% relative improvement in ALFWorld unseen success rate over standard OPD.
  arXiv  Detail & Related papers  (2026-06-17T23:58:14Z)
• On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agents [50.78688247330029]
  On-Policy Distillation (OPD) is a natural recipe for transferring such capabilities to smaller students.<n>We propose Guided On-Policy Distillation (Guided-OPD), a simple yet effective algorithm that mixes teacher- and student-generated turns within each rollout.<n>Guided-OPD improves Score by 21.1% and Success Rate by 25.5% over vanilla OPD on average.
  arXiv  Detail & Related papers  (2026-06-14T16:41:45Z)
• CLaaS: Continual learning as a service for sample efficient online learning [1.7646568737734192]
  We investigate an online continual learning setting in which agents learn from a stream of scenarios.<n>We propose continual learning as-a-service (CL), a system which enables agents to improve during deployment.<n>We evaluate CL on an adversarial task, demonstrating that parametric updates lead to superior forward transfer and less forgetting than in-context learning.
  arXiv  Detail & Related papers  (2026-06-04T01:14:19Z)
• Prune-OPD: Efficient and Reliable On-Policy Distillation for Long-Horizon Reasoning [66.52232008796294]
  Prune-OPD dynamically aligns training budgets with supervision quality.<n>It reduces training time by 37.6%--68.0% while preserving, and often improving, performance on challenging benchmarks.
  arXiv  Detail & Related papers  (2026-05-08T14:38:53Z)
• TCOD: Exploring Temporal Curriculum in On-Policy Distillation for Multi-turn Autonomous Agents [55.27396165691312]
  We identify a key limitation of vanilla OPD in multi-turn agent settings, which we term Trajectory-Level KL Instability.<n>We propose TCOD, a framework that controls the trajectory depth exposed to the student and progressively expands it with a curriculum schedule.<n> Experimental results across four student-teacher pairs show that TCOD mitigates KL escalation and enhances KL stability throughout training, improving agent performance by up to 18 points over vanilla OPD.
  arXiv  Detail & Related papers  (2026-04-27T03:38:27Z)
• Efficient Unsupervised Environment Design through Hierarchical Policy Representation Learning [28.99712640511788]
  Unsupervised Environment Design (UED) has emerged as a promising approach to developing general-purpose agents through automated curriculum.<n>We introduce a hierarchical Markov Decision Process (MDP) framework for environment design.<n>We show that our method outperforms baseline approaches while requiring fewer teacher-student interactions in a single episode.
  arXiv  Detail & Related papers  (2026-02-10T14:19:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.