Adaptive FastOPD: Progress-Aware Rollout Horizon Expansion for Efficient On-Policy Distillation

• URL: http://arxiv.org/abs/2607.29494v1
• Date: Fri, 31 Jul 2026 15:02:53 GMT
• Title: Adaptive FastOPD: Progress-Aware Rollout Horizon Expansion for Efficient On-Policy Distillation
• Abstract summary: On-policy distillation (OPD) provides dense teacher supervision along student-generated trajectories, but its online rollout process incurs substantial computational cost.<n>We propose Adaptive FastOPD, a progress-aware strategy that expands the rollout horizon only when learning near the current boundary region has plateaued.
• Score: 6.296437135279578
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: On-policy distillation (OPD) provides dense teacher supervision along student-generated trajectories, but its online rollout process incurs substantial computational cost, particularly when a few long responses delay batch completion. Existing acceleration methods typically control rollout length using fixed budgets or absolute teacher--student agreement thresholds, which may not reflect learning progress across different models and training stages. We propose Adaptive FastOPD, a progress-aware strategy that expands the rollout horizon only when learning near the current boundary region has plateaued and the current horizon is sufficiently utilized. The former is determined from four teacher--student signals measured relative to their values upon entering each horizon, making expansion responsive to stage-specific progress rather than a predefined step interval or an absolute threshold on the raw agreement signals, while the latter prevents a small number of long responses from triggering increases in rollout cost. Across two teacher--student pairs, Adaptive FastOPD achieves the highest average performance while reducing training time by 49.1--71.2\% relative to OPD 15K, and remains robust across a range of hyperparameter settings.

Related papers

• Know When to Stop, Where to Restart: Accelerating Multi-Turn Agentic On-Policy Distillation [45.25745441009372]
  Existing acceleration methods truncate or relocate the supervision signal according to fixed, offline budgets.<n>We propose STRIDE (Stop-and-Restart on-policy Distillation acceleration), which combines two complementary techniques.<n>Our method matches full-trajectory OPD and exceeds the 30B teacher at a $3.73times$ speedup, surpasses the baseline itself at $2.34times$, and retains a $4.51times$ speedup under cross-domain multi-teacher training.
  arXiv  Detail & Related papers  (2026-09-13T16:17:35Z)
• OPTD: On-Policy Transition Distillation with Consistency-Guided Adaptive Compression for Few-Step Diffusion Language Models [39.15861870749305]
  We propose OPTD, On-Policy Transition Distillation with consistency-guided adaptive compression.<n>It samples partial states from the few-step student's own trajectories, uses a frozen, question-only teacher to identify outcome-aligned future candidates, and orders them by current-state confidence.<n>It consistently improves the quality--efficiency trade-off and attains the strongest overall quality-constrained AUP among the evaluated few-step baselines.
  arXiv  Detail & Related papers  (2026-08-03T23:09:43Z)
• TurnOPD: Making On-Policy Distillation Turn-Aware for Efficient Long-Horizon Agent Training [52.6661804966136]
  On-policy distillation (OPD) trains a student policy by matching a stronger teacher on the student's own trajectories.<n>We propose TurnOPD, a turn-level budgeting strategy for efficient on-policy distillation of long-horizon agents.
  arXiv  Detail & Related papers  (2026-07-07T03:56:35Z)
• ATOD: Annealed Turn-aware On-policy Distillation for Multi-turn Autonomous Agents [6.4535223115254885]
  Training small language-model agents for long-horizon interactive tasks requires both fast imitation and reward-driven improvement.<n>We propose ATOD (Annealed Turn-aware On-policy Distillation), a hybrid online distillation algorithm that explicitly exploits this complementarity.
  arXiv  Detail & Related papers  (2026-06-26T07:56:32Z)
• Prefix-Guided On-Policy Distillation: Mining Golden Trajectories from Rollouts [48.550535291129584]
  Prefix-Guided On-Policy Distillation (PG-OPD) is a simple rollout-allocation framework that uses fixed-length prefixes to estimate trajectory value before expensive long-horizon generation.<n>Across diverse teacher-student combinations on AMC, AIME, and HMMT benchmarks, PG-OPD improves average accuracy by up to 4.80 points while reducing training time by up to 2.46x.
  arXiv  Detail & Related papers  (2026-06-20T11:18:34Z)
• Are Full Rollouts Necessary for On-Policy Distillation? [63.18243901591995]
  On-policy distillation (OPD) provides dense teacher feedback along student-generated rollouts.<n>We identify the rollout horizon as a key bottleneck in OPD that substantially impacts training efficiency.<n>We propose two simple horizon-control strategies: Progressive OPD, which gradually expands the rollout horizon during training, and Truncated OPD, which permanently performs distillation on reliable truncated rollouts.
  arXiv  Detail & Related papers  (2026-05-29T16:12:54Z)
• ADWIN: Adaptive Windows for Horizon-Aware On-Policy Distillation [11.916633988612439]
  On-policy distillation (OPD) transfers reasoning behavior by training a student on teacher feedback along student-generated trajectories.<n>We propose ADWIN, an adaptive-window framework for OPD that treats rollout length as an online admissibility decision.
  arXiv  Detail & Related papers  (2026-05-27T12:33:44Z)
• Learning to Foresee: Unveiling the Unlocking Efficiency of On-Policy Distillation [51.210887267509854]
  We argue that OPD's efficiency stems from a form of foresight'': it establishes a stable update trajectory toward the final model early in training.<n>We propose textbfEffOPD, a plug-and-play acceleration method that speeds up OPD by adaptively selecting an extrapolation step size and moving along the current update direction.
  arXiv  Detail & Related papers  (2026-05-12T08:19:15Z)
• SOPE: Stabilizing Off-Policy Evaluation for Online RL with Prior Data [49.370849653460716]
  We propose SOPE, an algorithm that uses an actor-aligned Off-Policy Policy Evaluation signal as an automated early-stopping mechanism.<n> evaluated on 25 continuous control tasks from the Minari benchmark suite.<n>SOPE improves baseline performance by up to 45.6% while reducing the required TFLOPs by up to 22x.
  arXiv  Detail & Related papers  (2026-05-07T08:32:09Z)
• Learning to Draft: Adaptive Speculative Decoding with Reinforcement Learning [67.88087883391475]
  We introduce Learning to Draft, a novel method that directly optimize for throughput of each draft-and-verify cycle.<n> LTD achieves speedup ratios ranging from 2.24x to 4.32x, outperforming the state-of-the-art method Eagle3 up to 36.4%.
  arXiv  Detail & Related papers  (2026-03-02T09:17:48Z)
• Steering Prototypes with Prompt-tuning for Rehearsal-free Continual Learning [47.83442130744575]
  Prototypes as representative class embeddings offer advantages in memory conservation and the mitigation of catastrophic forgetting. In this study, we introduce the Contrastive Prototypical Prompt ( CPP) approach. CPP achieves a significant 4% to 6% improvement over state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-03-16T16:23:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.