Prefix-Guided On-Policy Distillation: Mining Golden Trajectories from Rollouts
- URL: http://arxiv.org/abs/2606.21994v1
- Date: Sat, 20 Jun 2026 11:18:34 GMT
- Title: Prefix-Guided On-Policy Distillation: Mining Golden Trajectories from Rollouts
- Abstract summary: Prefix-Guided On-Policy Distillation (PG-OPD) is a simple rollout-allocation framework that uses fixed-length prefixes to estimate trajectory value before expensive long-horizon generation.<n>Across diverse teacher-student combinations on AMC, AIME, and HMMT benchmarks, PG-OPD improves average accuracy by up to 4.80 points while reducing training time by up to 2.46x.
- Score: 48.550535291129584
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: On-policy distillation (OPD) improves reasoning models by applying dense teacher supervision on student-sampled trajectories. However, scaling OPD to long-horizon mathematical reasoning exposes a reliability and efficiency problem: standard OPD assigns every sampled candidate the same long rollout budget, even though some trajectories may quickly become weakly aligned with the teacher and provide less useful supervision. Prior analyses suggest that successful OPD depends on local teacher-student compatibility, which can be measured by top-k overlap on student-visited prefixes. When this overlap is low, continuing to generate or train on long suffixes may waste computation and introduce noisy learning signal. To address this, we introduce Prefix-Guided On-Policy Distillation (PG-OPD), a simple rollout-allocation framework that uses fixed-length prefixes to estimate trajectory value before expensive long-horizon generation. PG-OPD first decodes every sampled candidate to the same prefix length, computes teacher-student top-k overlap within an early probe window of that prefix, and selectively continues high-overlap candidates to a fixed long length. Low-overlap candidates stop at the fixed prefix, avoiding unnecessary suffix generation. Across diverse teacher-student combinations on AMC, AIME, and HMMT benchmarks, PG-OPD improves average accuracy by up to 4.80 points while reducing training time by up to 2.46x. These results suggest that prefix-level compatibility provides a practical signal for directing OPD computation toward trajectories that remain learnable from the teacher.
Related papers
- ReOrder-OPD:Reliability-Aware Prompt Ordering for On-Policy Distillation [25.69241346799495]
We define prompt-level teacher continuation reliability $R$ as the teacher's probability of reaching a correct answer from a student prefix.<n>We show that high-$R$ prompts yield larger OPD gains and that descending-$R$ training outperforms random and ascending orders.
arXiv Detail & Related papers (2026-08-11T13:27:56Z) - Multi-Turn On-Policy Distillation with Prefix Replay [73.10000453999088]
We propose Replayed-Prefix On-Policy Distillation (ReOPD) for agentic tasks.<n>ReOPD reuses pre-collected teacher trajectories as replayed prefixes.<n>It preserves or improves OPD-level accuracy, uses zero tool calls during student training, and is at least 4$times$ faster per training step than OPD.
arXiv Detail & Related papers (2026-07-06T07:56:53Z) - Trajectory-Refined Distillation [25.346810464266497]
Trajectory-Refined Distillation (TRD) is a trajectory-level correction method that revises the student's rollout under the teacher guidance while within on-policy support.<n> TRD can also be applied to on-policy self-distillation (OPSD), a parameter-sharing variant that uses the student model conditioned on privileged informations as the teacher.
arXiv Detail & Related papers (2026-06-07T03:17:25Z) - ADWIN: Adaptive Windows for Horizon-Aware On-Policy Distillation [11.916633988612439]
On-policy distillation (OPD) transfers reasoning behavior by training a student on teacher feedback along student-generated trajectories.<n>We propose ADWIN, an adaptive-window framework for OPD that treats rollout length as an online admissibility decision.
arXiv Detail & Related papers (2026-05-27T12:33:44Z) - Prefix Teach, Suffix Fade: Local Teachability Collapse in Strong-to-Weak On-Policy Distillation [49.117085054884676]
On-policy distillation trains a student model on its own rollouts using dense feedback from a stronger teacher.<n>We operationalize this principle through a trajectory-specific release rule.<n> Experimental results across strong-to-weak distillation tasks indicate that this release rule consistently outperforms standard full-trajectory OPD.
arXiv Detail & Related papers (2026-05-13T15:05:30Z) - On-Policy Distillation with Best-of-N Teacher Rollout Selection [54.91780727674628]
We propose BRTS, a Best-of-N Rollout Teacher Selection framework for on-policy distillation.<n>BRTS augments standard student-context OPD with a teacher-context supervision branch constructed from the teacher trajectory.<n>BRTS improves over standard OPD on challenging reasoning benchmarks, with the largest gains on harder datasets.
arXiv Detail & Related papers (2026-05-10T19:49:00Z) - Prune-OPD: Efficient and Reliable On-Policy Distillation for Long-Horizon Reasoning [66.52232008796294]
Prune-OPD dynamically aligns training budgets with supervision quality.<n>It reduces training time by 37.6%--68.0% while preserving, and often improving, performance on challenging benchmarks.
arXiv Detail & Related papers (2026-05-08T14:38:53Z) - Long-Chain Reasoning Distillation via Adaptive Prefix Alignment [57.130176131042965]
We propose a framework that exploits teacher CoTs for distillation through adaptive prefix alignment.<n>P-ALIGN adaptively truncates teacher-generated reasoning trajectories by determining whether the remaining suffix is concise.<n>Experiments on multiple mathematical reasoning benchmarks demonstrate that P-ALIGN outperforms all baselines by over 3%.
arXiv Detail & Related papers (2026-01-15T04:40:45Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.