Pass the Baton: Trajectory-Relayed On-Policy Distillation

• URL: http://arxiv.org/abs/2607.26057v1
• Date: Tue, 28 Jul 2026 17:59:46 GMT
• Title: Pass the Baton: Trajectory-Relayed On-Policy Distillation
• Abstract summary: On-policy distillation grounds token-level supervision in the student's own trajectory, yet suffers from prefix failure.<n>We identify a teacher-student continuation asymmetry on failed prefixes, and convert it into a label-free handoff trigger in Relay On-Policy Distillation (Relay-OPD)<n>During training, Relay-OPD constructs relay trajectories by letting the teacher briefly take over at detected trigger points to produce a teacher leg, after which the student resumes and is optimized on the resulting trajectory.
• Score: 23.50689651536186
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: On-policy distillation (OPD) grounds token-level supervision in the student's own trajectory, yet suffers from prefix failure: once the student commits to a wrong reasoning direction, all subsequent generation builds on this deviation, producing misdirected continuations that elicit unreliable supervision and waste compute. We identify a teacher-student continuation asymmetry on failed prefixes, where the teacher tends to redirect while the student continues along the original direction, and convert it into a label-free handoff trigger in Relay On-Policy Distillation (Relay-OPD). During training, Relay-OPD constructs relay trajectories by letting the teacher briefly take over at detected trigger points to produce a teacher leg, after which the student resumes and is optimized on the resulting trajectory. A limited relay budget concentrates intervention on critical early positions while limiting departure from the student policy. With a Qwen3-4B-Instruct-2507 teacher and Qwen3-0.6B/1.7B-Non-Thinking students on eight mathematical reasoning benchmarks, Relay-OPD achieves the best or second-best results on every benchmark, outperforming standard OPD by +5.73% and the strongest baseline FastOPD by +1.49% on average for 1.7B, with consistent gains at 0.6B. Training trajectory length is reduced by over 50%.

Related papers

• Verify Before You Distill: Prompt-Level Teacher Gating for On-Policy Distillation [37.73879161170101]
  On-policy distillation (OPD) accelerates post-training by providing dense token-level supervision from a frozen teacher on the student's own rollouts.<n> Vanilla OPD applies this supervision uniformly across prompts, without checking whether the teacher is reliable for each prompt.<n>We introduce Teacher-Gated On-Policy Distillation (TGOPD), built on the principle that teacher reliability should be verified at the prompt level before dense supervision is admitted.
  arXiv  Detail & Related papers  (2026-09-02T17:54:09Z)
• On-Policy Distillation Meets Off-Policy GRPO: Training Compact Instruction-Following Rerankers [0.0]
  We revisit reranker distillation through the lens of reinforcement learning.<n>We propose a two-stage framework combining off-policy teacher optimization with on-policy student distillation.
  arXiv  Detail & Related papers  (2026-09-01T23:36:19Z)
• Zone of Proximal Policy Optimization: Teacher in Prompts, Not Gradients [89.89504265663057]
  Zone of Proximal Policy Optimization (ZPPO) is inspired by Vygotsky's zone of proximal development.<n>ZPPO pairs one correct teacher response with one incorrect student response as anonymized candidates the student must discriminate.<n>A prompt replay buffer recirculates each hard question until it either graduates-the student's mean rollout accuracy on it reaches half- or is FIFO-evicted.
  arXiv  Detail & Related papers  (2026-06-16T17:46:02Z)
• On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agents [50.78688247330029]
  On-Policy Distillation (OPD) is a natural recipe for transferring such capabilities to smaller students.<n>We propose Guided On-Policy Distillation (Guided-OPD), a simple yet effective algorithm that mixes teacher- and student-generated turns within each rollout.<n>Guided-OPD improves Score by 21.1% and Success Rate by 25.5% over vanilla OPD on average.
  arXiv  Detail & Related papers  (2026-06-14T16:41:45Z)
• Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation [81.10000755917712]
  On-policy distillation transfers reasoning capabilities by training a student model on its own generated trajectories using token-level feedback from a teacher.<n>As student-generated prefixes lengthen, the teacher's next-token distribution becomes less confident and less discriminative.<n>To mitigate SFD, we introduce textbfLookahead Group Reward (ours).
  arXiv  Detail & Related papers  (2026-05-29T04:39:20Z)
• Prefix Teach, Suffix Fade: Local Teachability Collapse in Strong-to-Weak On-Policy Distillation [49.117085054884676]
  On-policy distillation trains a student model on its own rollouts using dense feedback from a stronger teacher.<n>We operationalize this principle through a trajectory-specific release rule.<n> Experimental results across strong-to-weak distillation tasks indicate that this release rule consistently outperforms standard full-trajectory OPD.
  arXiv  Detail & Related papers  (2026-05-13T15:05:30Z)
• On-Policy Distillation with Best-of-N Teacher Rollout Selection [54.91780727674628]
  We propose BRTS, a Best-of-N Rollout Teacher Selection framework for on-policy distillation.<n>BRTS augments standard student-context OPD with a teacher-context supervision branch constructed from the teacher trajectory.<n>BRTS improves over standard OPD on challenging reasoning benchmarks, with the largest gains on harder datasets.
  arXiv  Detail & Related papers  (2026-05-10T19:49:00Z)
• Prune-OPD: Efficient and Reliable On-Policy Distillation for Long-Horizon Reasoning [66.52232008796294]
  Prune-OPD dynamically aligns training budgets with supervision quality.<n>It reduces training time by 37.6%--68.0% while preserving, and often improving, performance on challenging benchmarks.
  arXiv  Detail & Related papers  (2026-05-08T14:38:53Z)
• MAD-OPD: Breaking the Ceiling in On-Policy Distillation via Multi-Agent Debate [17.31967380175203]
  We propose MAD-OPD (Multi-Agent Debate-driven On-Policy Distillation), which recasts the distillation teacher as a deliberative collective of teachers that debate over the student's on-policy state.<n>We also introduce On-Policy Agentic Distillation (OPAD), which adds step-level sampling to stabilize training under multi-step error compounding.
  arXiv  Detail & Related papers  (2026-05-02T09:41:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.