ReOrder-OPD:Reliability-Aware Prompt Ordering for On-Policy Distillation
- URL: http://arxiv.org/abs/2608.10905v1
- Date: Tue, 11 Aug 2026 13:27:56 GMT
- Title: ReOrder-OPD:Reliability-Aware Prompt Ordering for On-Policy Distillation
- Abstract summary: We define prompt-level teacher continuation reliability $R$ as the teacher's probability of reaching a correct answer from a student prefix.<n>We show that high-$R$ prompts yield larger OPD gains and that descending-$R$ training outperforms random and ascending orders.
- Score: 25.69241346799495
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: On-policy distillation (OPD) applies token-level teacher supervision to student-generated trajectories, but this supervision is not always reliable. Existing methods use local confidence or teacher-student agreement to weight, filter, or truncate the sampled trajectory. These signals do not directly determine whether the teacher can continue a student prefix to a correct answer, and trajectory-level interventions can conflate one rollout's unreliability with low expected training value of its prompt. We define prompt-level teacher continuation reliability $R$ as the teacher's probability of reaching a correct answer from a student prefix, averaged over prefixes and trajectories induced by the current student. Oracle experiments show that high-$R$ prompts yield larger OPD gains and that descending-$R$ training outperforms random and ascending orders on a fixed prompt pool. Because estimating $R$ requires many teacher continuations, we use the maximum ROUGE-5 F1 between one independent student rollout and verifier-correct same-prompt teacher trajectories. Across ten equal-frequency bins of this actual score, mean $R$ rises monotonically, showing that the proxy separates coarse reliability levels. ReOrder-OPD sorts prompts by the proxy, then draws independent on-policy training trajectories for vanilla OPD. It improves every matched aggregate comparison across Qwen3 and Gemma4 mathematics settings and Qwen3 code settings. Gains in all six FiRe-OPD and ExOPD settings show that prompt ordering complements within-trajectory supervision.
Related papers
- Verify Before You Distill: Prompt-Level Teacher Gating for On-Policy Distillation [37.73879161170101]
On-policy distillation (OPD) accelerates post-training by providing dense token-level supervision from a frozen teacher on the student's own rollouts.<n> Vanilla OPD applies this supervision uniformly across prompts, without checking whether the teacher is reliable for each prompt.<n>We introduce Teacher-Gated On-Policy Distillation (TGOPD), built on the principle that teacher reliability should be verified at the prompt level before dense supervision is admitted.
arXiv Detail & Related papers (2026-09-02T17:54:09Z) - REOPD: Reliability-Adaptive Reward Extrapolation for On-Policy Distillation [10.492445451268727]
REOPD is a reliability-adaptive reward extrapolation framework for on-policy distillation.<n>It preserves teacher alignment while selectively extrapolating along reliable teacher-reference directions.
arXiv Detail & Related papers (2026-08-12T06:15:33Z) - H$^2$SD: Hybrid Hindsight Self-Distillation [61.71131241473028]
Reinforcement learning with verifiable rewards (RLVR) provides reliable outcome supervision for language model reasoning.<n>Existing self-distillation methods add a privileged teacher but typically assign it a fixed role.<n>We introduce Hybrid Hindsight Self-Distillation ($mathrmH2mathrmSD$), which jointly adapts teacher context and update strategy to trajectory correctness.
arXiv Detail & Related papers (2026-07-21T10:47:27Z) - Multi-Turn On-Policy Distillation with Prefix Replay [73.10000453999088]
We propose Replayed-Prefix On-Policy Distillation (ReOPD) for agentic tasks.<n>ReOPD reuses pre-collected teacher trajectories as replayed prefixes.<n>It preserves or improves OPD-level accuracy, uses zero tool calls during student training, and is at least 4$times$ faster per training step than OPD.
arXiv Detail & Related papers (2026-07-06T07:56:53Z) - Prefix-Guided On-Policy Distillation: Mining Golden Trajectories from Rollouts [48.550535291129584]
Prefix-Guided On-Policy Distillation (PG-OPD) is a simple rollout-allocation framework that uses fixed-length prefixes to estimate trajectory value before expensive long-horizon generation.<n>Across diverse teacher-student combinations on AMC, AIME, and HMMT benchmarks, PG-OPD improves average accuracy by up to 4.80 points while reducing training time by up to 2.46x.
arXiv Detail & Related papers (2026-06-20T11:18:34Z) - Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation [81.10000755917712]
On-policy distillation transfers reasoning capabilities by training a student model on its own generated trajectories using token-level feedback from a teacher.<n>As student-generated prefixes lengthen, the teacher's next-token distribution becomes less confident and less discriminative.<n>To mitigate SFD, we introduce textbfLookahead Group Reward (ours).
arXiv Detail & Related papers (2026-05-29T04:39:20Z) - On-Policy Distillation with Best-of-N Teacher Rollout Selection [54.91780727674628]
We propose BRTS, a Best-of-N Rollout Teacher Selection framework for on-policy distillation.<n>BRTS augments standard student-context OPD with a teacher-context supervision branch constructed from the teacher trajectory.<n>BRTS improves over standard OPD on challenging reasoning benchmarks, with the largest gains on harder datasets.
arXiv Detail & Related papers (2026-05-10T19:49:00Z) - Prune-OPD: Efficient and Reliable On-Policy Distillation for Long-Horizon Reasoning [66.52232008796294]
Prune-OPD dynamically aligns training budgets with supervision quality.<n>It reduces training time by 37.6%--68.0% while preserving, and often improving, performance on challenging benchmarks.
arXiv Detail & Related papers (2026-05-08T14:38:53Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.