ADWIN: Adaptive Windows for Horizon-Aware On-Policy Distillation

• URL: http://arxiv.org/abs/2605.28396v1
• Date: Wed, 27 May 2026 12:33:44 GMT
• Title: ADWIN: Adaptive Windows for Horizon-Aware On-Policy Distillation
• Abstract summary: On-policy distillation (OPD) transfers reasoning behavior by training a student on teacher feedback along student-generated trajectories.<n>We propose ADWIN, an adaptive-window framework for OPD that treats rollout length as an online admissibility decision.
• Score: 11.916633988612439
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: On-policy distillation (OPD) transfers reasoning behavior by training a student on teacher feedback along student-generated trajectories, but standard full-rollout training ties every update to a costly completion and can over-allocate supervision to late positions with low marginal value for the current student. We revisit this assumption through the useful supervision horizon: student-induced rollouts can drift from teacher-preferred continuations, while aligned prefixes may already preserve the long-horizon OPD update direction. We propose ADWIN, an adaptive-window framework for OPD that treats rollout length as an online admissibility decision, training on short teacher-anchored prefixes while using delayed full-rollout probes to audit prefix--full alignment and adapt the next horizon with staleness control. Across math and code reasoning benchmarks in single-task, multi-task, and strong-to-weak settings, ADWIN improves the accuracy--compute trade-off over full-rollout OPD and prefix-based baselines, reducing end-to-end training cost by up to 4.1 times while achieving comparable or better accuracy.

Related papers

• Adaptive FastOPD: Progress-Aware Rollout Horizon Expansion for Efficient On-Policy Distillation [6.296437135279578]
  On-policy distillation (OPD) provides dense teacher supervision along student-generated trajectories, but its online rollout process incurs substantial computational cost.<n>We propose Adaptive FastOPD, a progress-aware strategy that expands the rollout horizon only when learning near the current boundary region has plateaued.
  arXiv  Detail & Related papers  (2026-07-31T15:02:53Z)
• TurnOPD: Making On-Policy Distillation Turn-Aware for Efficient Long-Horizon Agent Training [52.6661804966136]
  On-policy distillation (OPD) trains a student policy by matching a stronger teacher on the student's own trajectories.<n>We propose TurnOPD, a turn-level budgeting strategy for efficient on-policy distillation of long-horizon agents.
  arXiv  Detail & Related papers  (2026-07-07T03:56:35Z)
• Multi-Turn On-Policy Distillation with Prefix Replay [73.10000453999088]
  We propose Replayed-Prefix On-Policy Distillation (ReOPD) for agentic tasks.<n>ReOPD reuses pre-collected teacher trajectories as replayed prefixes.<n>It preserves or improves OPD-level accuracy, uses zero tool calls during student training, and is at least 4$times$ faster per training step than OPD.
  arXiv  Detail & Related papers  (2026-07-06T07:56:53Z)
• Prefix-Guided On-Policy Distillation: Mining Golden Trajectories from Rollouts [48.550535291129584]
  Prefix-Guided On-Policy Distillation (PG-OPD) is a simple rollout-allocation framework that uses fixed-length prefixes to estimate trajectory value before expensive long-horizon generation.<n>Across diverse teacher-student combinations on AMC, AIME, and HMMT benchmarks, PG-OPD improves average accuracy by up to 4.80 points while reducing training time by up to 2.46x.
  arXiv  Detail & Related papers  (2026-06-20T11:18:34Z)
• Trajectory-Refined Distillation [25.346810464266497]
  Trajectory-Refined Distillation (TRD) is a trajectory-level correction method that revises the student's rollout under the teacher guidance while within on-policy support.<n> TRD can also be applied to on-policy self-distillation (OPSD), a parameter-sharing variant that uses the student model conditioned on privileged informations as the teacher.
  arXiv  Detail & Related papers  (2026-06-07T03:17:25Z)
• Are Full Rollouts Necessary for On-Policy Distillation? [63.18243901591995]
  On-policy distillation (OPD) provides dense teacher feedback along student-generated rollouts.<n>We identify the rollout horizon as a key bottleneck in OPD that substantially impacts training efficiency.<n>We propose two simple horizon-control strategies: Progressive OPD, which gradually expands the rollout horizon during training, and Truncated OPD, which permanently performs distillation on reliable truncated rollouts.
  arXiv  Detail & Related papers  (2026-05-29T16:12:54Z)
• Distribution Corrected Offline Data Distillation for Large Language Models [8.778789513095907]
  Distilling reasoning traces from strong large language models into smaller ones is a promising route to improve intelligence in resource-constrained settings.<n>Existing approaches face a fundamental trade-off: offline distillation from teacher-generated traces provides high-quality, sample-efficient supervision but suffers from distributional drift.<n>We propose a principled offline reasoning distillation framework that preserves the efficiency and supervision quality of offline teacher-generated data while correcting teacher-student distribution drift.
  arXiv  Detail & Related papers  (2026-05-13T19:47:31Z)
• Prefix Teach, Suffix Fade: Local Teachability Collapse in Strong-to-Weak On-Policy Distillation [49.117085054884676]
  On-policy distillation trains a student model on its own rollouts using dense feedback from a stronger teacher.<n>We operationalize this principle through a trajectory-specific release rule.<n> Experimental results across strong-to-weak distillation tasks indicate that this release rule consistently outperforms standard full-trajectory OPD.
  arXiv  Detail & Related papers  (2026-05-13T15:05:30Z)
• On-Policy Distillation with Best-of-N Teacher Rollout Selection [54.91780727674628]
  We propose BRTS, a Best-of-N Rollout Teacher Selection framework for on-policy distillation.<n>BRTS augments standard student-context OPD with a teacher-context supervision branch constructed from the teacher trajectory.<n>BRTS improves over standard OPD on challenging reasoning benchmarks, with the largest gains on harder datasets.
  arXiv  Detail & Related papers  (2026-05-10T19:49:00Z)
• Prune-OPD: Efficient and Reliable On-Policy Distillation for Long-Horizon Reasoning [66.52232008796294]
  Prune-OPD dynamically aligns training budgets with supervision quality.<n>It reduces training time by 37.6%--68.0% while preserving, and often improving, performance on challenging benchmarks.
  arXiv  Detail & Related papers  (2026-05-08T14:38:53Z)
• TCOD: Exploring Temporal Curriculum in On-Policy Distillation for Multi-turn Autonomous Agents [55.27396165691312]
  We identify a key limitation of vanilla OPD in multi-turn agent settings, which we term Trajectory-Level KL Instability.<n>We propose TCOD, a framework that controls the trajectory depth exposed to the student and progressively expands it with a curriculum schedule.<n> Experimental results across four student-teacher pairs show that TCOD mitigates KL escalation and enhances KL stability throughout training, improving agent performance by up to 18 points over vanilla OPD.
  arXiv  Detail & Related papers  (2026-04-27T03:38:27Z)
• Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillation [7.2992280064983825]
  On-policy distillation (OPD) has emerged as an efficient post-training paradigm for large language models.<n>Standard OPD requires a live teacher inference server throughout training, resulting in substantial infrastructure overhead.<n>We propose Lightning OPD, an offline on-policy distillation framework that enforces teacher consistency by precomputing teacher log-probabilities over SFT rollouts.
  arXiv  Detail & Related papers  (2026-04-14T17:44:50Z)
• Beat the long tail: Distribution-Aware Speculative Decoding for RL Training [75.75462952580796]
  We propose a Distribution Aware Speculative decoding framework that accelerates RL rollouts without altering model outputs.<n>Experiments on math and code reasoning tasks show that DAS reduces rollout time up to 50% while preserving identical training curves.
  arXiv  Detail & Related papers  (2025-11-17T19:02:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.