Know When to Stop, Where to Restart: Accelerating Multi-Turn Agentic On-Policy Distillation

• URL: http://arxiv.org/abs/2609.14636v1
• Date: Sun, 13 Sep 2026 16:17:35 GMT
• Title: Know When to Stop, Where to Restart: Accelerating Multi-Turn Agentic On-Policy Distillation
• Abstract summary: Existing acceleration methods truncate or relocate the supervision signal according to fixed, offline budgets.<n>We propose STRIDE (Stop-and-Restart on-policy Distillation acceleration), which combines two complementary techniques.<n>Our method matches full-trajectory OPD and exceeds the 30B teacher at a $3.73times$ speedup, surpasses the baseline itself at $2.34times$, and retains a $4.51times$ speedup under cross-domain multi-teacher training.
• Score: 45.25745441009372
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: On-policy distillation (OPD) has become a standard approach for transferring capabilities from large teachers to compact students. Its cost, however, is dominated by autoregressive student rollouts and scales poorly in multi-turn agentic settings. Existing acceleration methods truncate or relocate the supervision signal according to fixed, offline budgets, despite substantial variation in teacher-signal reliability both within and across trajectories. Our empirical analysis on $τ^2$-bench reveals a clear structure in this variation: informative supervision is concentrated in the prefix of each turn, and, most importantly for multi-turn agentic training, the cross-turn loss of teacher endorsement is temporally locked to the student's first erroneous action rather than accumulating gradually over turns. Building on these findings, we propose STRIDE (Stop-and-Restart on-policy Distillation acceleration), which combines two complementary techniques: adaptive early stopping, which terminates a rollout once the cumulative teacher log-probability falls below an out-of-distribution threshold, and a prefix buffer, which caches high-quality prefixes and restarts generation at the weakest correct turn. Together, these mechanisms induce a data-driven curriculum that progressively extends coverage to later turns. On $τ^2$-bench retail, our method matches full-trajectory OPD and exceeds the 30B teacher at a $3.73\times$ speedup, surpasses the baseline itself at $2.34\times$, and retains a $4.51\times$ speedup under cross-domain multi-teacher training. As a supplementary generalization test beyond the agentic setting, STRIDE outperforms full OPD on AIME 2025 at a $5.10\times$ speedup and on AIME 2024 at a $3.08\times$ speedup; averaged across the two evaluations, both fixed-budget truncation baselines remain below full OPD.

Related papers

• D$^3$-MOPD: Adaptive Dynamic Domain ScheDuling for Efficient Multi-Teacher Distillation [50.89072318748398]
  Multi-teacher on-policy distillation distills several domain-expert teachers into a single student.<n>Existing approaches typically fix the per-domain data mixture before training.<n>A fixed mixture wastes compute on fast-converging domains and undertrains slower-converging ones.<n>We propose D$3$-MOPD, a zero-overhead scheduler that repurposes the per-domain reverse-KL signal already produced during training to adapt the domain mixture online.
  arXiv  Detail & Related papers  (2026-08-25T17:57:11Z)
• Adaptive FastOPD: Progress-Aware Rollout Horizon Expansion for Efficient On-Policy Distillation [6.296437135279578]
  On-policy distillation (OPD) provides dense teacher supervision along student-generated trajectories, but its online rollout process incurs substantial computational cost.<n>We propose Adaptive FastOPD, a progress-aware strategy that expands the rollout horizon only when learning near the current boundary region has plateaued.
  arXiv  Detail & Related papers  (2026-07-31T15:02:53Z)
• DASH-OPD: Discrepancy-Aware Switching with Hysteresis for On-Policy Distillation [9.897015521661915]
  On-policy distillation (OPD) trains student models on their own rollouts to reduce exposure bias.<n>Existing curriculum learning methods regulate how much teacher support is used according to training progress, but cannot determine when it is needed.<n>We propose DASH-OPD, a new agentic OPD method that can switch executors adaptively and bidirectionally.
  arXiv  Detail & Related papers  (2026-07-31T06:57:17Z)
• TurnOPD: Making On-Policy Distillation Turn-Aware for Efficient Long-Horizon Agent Training [52.6661804966136]
  On-policy distillation (OPD) trains a student policy by matching a stronger teacher on the student's own trajectories.<n>We propose TurnOPD, a turn-level budgeting strategy for efficient on-policy distillation of long-horizon agents.
  arXiv  Detail & Related papers  (2026-07-07T03:56:35Z)
• ATOD: Annealed Turn-aware On-policy Distillation for Multi-turn Autonomous Agents [6.4535223115254885]
  Training small language-model agents for long-horizon interactive tasks requires both fast imitation and reward-driven improvement.<n>We propose ATOD (Annealed Turn-aware On-policy Distillation), a hybrid online distillation algorithm that explicitly exploits this complementarity.
  arXiv  Detail & Related papers  (2026-06-26T07:56:32Z)
• On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agents [50.78688247330029]
  On-Policy Distillation (OPD) is a natural recipe for transferring such capabilities to smaller students.<n>We propose Guided On-Policy Distillation (Guided-OPD), a simple yet effective algorithm that mixes teacher- and student-generated turns within each rollout.<n>Guided-OPD improves Score by 21.1% and Success Rate by 25.5% over vanilla OPD on average.
  arXiv  Detail & Related papers  (2026-06-14T16:41:45Z)
• Are Full Rollouts Necessary for On-Policy Distillation? [63.18243901591995]
  On-policy distillation (OPD) provides dense teacher feedback along student-generated rollouts.<n>We identify the rollout horizon as a key bottleneck in OPD that substantially impacts training efficiency.<n>We propose two simple horizon-control strategies: Progressive OPD, which gradually expands the rollout horizon during training, and Truncated OPD, which permanently performs distillation on reliable truncated rollouts.
  arXiv  Detail & Related papers  (2026-05-29T16:12:54Z)
• DUET: Optimize Token-Budget Allocation for Reinforcement Learning with Verifiable Rewards [37.28110997883518]
  Reinforcement learning with verifiable rewards generates hundreds of thousands of tokens per training step.<n>We show that jointly tuning both decisions under a shared compute budget improves both reasoning quality and wall-clock training time.
  arXiv  Detail & Related papers  (2026-05-08T20:03:19Z)
• TCOD: Exploring Temporal Curriculum in On-Policy Distillation for Multi-turn Autonomous Agents [55.27396165691312]
  We identify a key limitation of vanilla OPD in multi-turn agent settings, which we term Trajectory-Level KL Instability.<n>We propose TCOD, a framework that controls the trajectory depth exposed to the student and progressively expands it with a curriculum schedule.<n> Experimental results across four student-teacher pairs show that TCOD mitigates KL escalation and enhances KL stability throughout training, improving agent performance by up to 18 points over vanilla OPD.
  arXiv  Detail & Related papers  (2026-04-27T03:38:27Z)
• Near-Future Policy Optimization [51.760544033045726]
  Reinforcement learning with verifiable rewards (RLVR) has become a core post-training recipe.<n>We propose textbfNear-Future textbfPolicy textbfOptimization (textbfNPO), a simple mixed-policy scheme that learns from a policy's own near-future self.
  arXiv  Detail & Related papers  (2026-04-22T16:20:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.