Bridging Reasoning Trajectories in On-Policy Distillation via Near-Future Guidance

• URL: http://arxiv.org/abs/2606.00305v1
• Date: Fri, 29 May 2026 19:32:07 GMT
• Title: Bridging Reasoning Trajectories in On-Policy Distillation via Near-Future Guidance
• Abstract summary: On-Policy Distillation improves large language model reasoning by training a student model on trajectories sampled from its own policy under teacher supervision.<n>We show that this "trajectory-sampled but token-learned" mechanism cannot reliably bridge student trajectories toward teacher trajectories.<n>We propose Trajectory-aware OPD, which uses near-future trajectory information to identify real divergent states and distribute guidance across multiple future tokens.
• Score: 10.04930078540686
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: On-Policy Distillation (OPD) improves large language model reasoning by training a student model on trajectories sampled from its own policy under teacher supervision. Although OPD operates on trajectories, its learning signal remains token-level: it identifies deviations through high-loss tokens and repairs them through local reverse-KL correction. We show that this "trajectory-sampled but token-learned" mechanism cannot reliably bridge student trajectories toward teacher trajectories. About 30% of high-loss tokens fall into the low-divergence regime, indicating that many are surface-form mismatches rather than real reasoning forks. Moreover, even truly divergent tokens are difficult to repair with isolated token-level supervision, since reasoning failures often unfold as short-horizon distributional drift. We propose Trajectory-aware OPD (TOPD), which uses near-future trajectory information to identify real divergent states and distribute guidance across multiple future tokens. Experiments show that suppressing non-divergent high-loss tokens improves standard OPD from 47.8% to 48.2% average accuracy, while TOPD further improves performance to 52.2%, with gains on AIME24 from 60.0% to 63.3% and AIME25 from 46.7% to 53.3%.

Related papers

• Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement [27.122890248991556]
  On-policy distillation (OPD) offers dense token-level supervision as an alternative to the sparse outcome-level advantages of reinforcement learning with verifiable rewards (RLVR)<n>We quantitatively analyze teacher supervision during OPD training and find substantial noise whose prevalence increases with teacher scale.<n>We find that learning concentrates on low log-probability tokens, and using a single fixed negative advantage matches the performance of teacher-provided ones.
  arXiv  Detail & Related papers  (2026-08-31T16:22:10Z)
• TTPO: Test-Time Policy Optimization [53.81524570216593]
  Test-Time Policy Optimization (TTPO) is an asymmetric objective that distills agreeing rollouts via OPSD and penalizes disagreeing rollouts with Grouped RL.<n>We show that TTPO matches label-supervised OPSD on five competition-level benchmarks, raises Qwen3-1.7B from 38.0% to 45.2% in TTT, yields +25.2% to +36.4% without thinking, and shows strong cross-task generalization.
  arXiv  Detail & Related papers  (2026-08-27T17:58:10Z)
• On-Policy Self-Distillation in Diffusion Models [89.32656931795293]
  Reinforcement learning can align diffusion models with human preferences and task-specific objectives, but endpoint rewards do not specify how an intermediate denoising prediction should change.<n>We introduce DiffusionOPSD as an on-policy self-distillation framework that converts image-level reward guidance into explicit targets for clean-output predictions at sampled queries.<n>Across SD 3.5-M and the step-distilled Z-Image-Turbo, our approach achieves the best final held-out scores in 19 of 20 reward-matched settings across two backbones and ten evaluators.
  arXiv  Detail & Related papers  (2026-08-25T14:55:24Z)
• Distill Where You Fail: Recovering Learning Signals of Negative RL-Groups from Adaptive Teacher Guidance [50.77757355236912]
  We propose RSTG (Recovering Learning Signals via Adaptive Teacher Guidance), which applies distillation selectively and precisely where it matters most.<n> Experiments demonstrate that RSTG substantially outperforms naive GRPO+OPD by +4.02% on math and+3.05% on code.
  arXiv  Detail & Related papers  (2026-08-01T17:29:14Z)
• Outcome-Confounded Local Supervision in On-Policy Distillation [0.5937989874256571]
  On-policy distillation trains a student on its own trajectories while a teacher supplies dense token-level likelihoods at student-visited prefixes.<n>We show that both readings are confounded by the outcome of the completed trajectory.<n>We introduce an outcome-resolved diagnostic that crosses pointwise teacher-student divergence with final-answer correctness.
  arXiv  Detail & Related papers  (2026-07-26T16:03:33Z)
• H$^2$SD: Hybrid Hindsight Self-Distillation [61.71131241473028]
  Reinforcement learning with verifiable rewards (RLVR) provides reliable outcome supervision for language model reasoning.<n>Existing self-distillation methods add a privileged teacher but typically assign it a fixed role.<n>We introduce Hybrid Hindsight Self-Distillation ($mathrmH2mathrmSD$), which jointly adapts teacher context and update strategy to trajectory correctness.
  arXiv  Detail & Related papers  (2026-07-21T10:47:27Z)
• Escaping the KL Agreement Trap in On-Policy Distillation [23.908556717663057]
  We propose an online OPD termination rule that detects persistent low-KL agreement with a dynamic training-adaptive threshold.<n>KAT improves avg@k accuracy by 2.66% and pass@k by 3.43% across four mathematical benchmarks, while reducing average rollout length by 59.73%.
  arXiv  Detail & Related papers  (2026-06-08T13:28:54Z)
• Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation [81.10000755917712]
  On-policy distillation transfers reasoning capabilities by training a student model on its own generated trajectories using token-level feedback from a teacher.<n>As student-generated prefixes lengthen, the teacher's next-token distribution becomes less confident and less discriminative.<n>To mitigate SFD, we introduce textbfLookahead Group Reward (ours).
  arXiv  Detail & Related papers  (2026-05-29T04:39:20Z)
• ESPO: Early-Stopping Proximal Policy Optimization [78.79610718910628]
  ESPO (Early-Stopping Proximal Policy Optimization) detects trajectory failure on-the-fly and terminates rollouts early.<n>On DeepSeek-R1-Distill-Qwen-7B trained for mathematical reasoning, ESPO surpasses PPO on AIME2024 (46.28% vs. 45.25%), AMC2023 (85.83% vs. 82.94%), and MATH-500 (87.42% vs. 85.43%)
  arXiv  Detail & Related papers  (2026-05-28T12:40:22Z)
• CEPO: RLVR Self-Distillation using Contrastive Evidence Policy Optimization [50.59956036193097]
  A model produces a correct solution under reinforcement learning with verifiable rewards (RLVR)<n>Every token receives the same reward signal regardless of whether it was a decisive reasoning step or a grammatical filler.<n>We propose Contrastive Evidence Policy Optimization (CEPO)<n>CEPO asks a sharper question at every token: "does the correct answer favor this token?" but "does the correct answer favor it while the wrong answer disfavors it?"
  arXiv  Detail & Related papers  (2026-05-19T06:46:19Z)
• Revisiting DAgger in the Era of LLM-Agents [35.615579397673166]
  Long-horizon LM agents learn from multi-turn interaction, where a single early mistake can alter the subsequent state distribution and derail the whole trajectory.<n>Existing recipes fall short in complementary ways: supervised fine-tuning provides dense teacher supervision, while reinforcement learning with verifiable rewards avoids this off-policy mismatch.<n>We address this dilemma by revisiting the dataset aggregation (DAgger) for multi-turn LM agents.
  arXiv  Detail & Related papers  (2026-05-13T02:40:28Z)
• SOD: Step-wise On-policy Distillation for Small Language Model Agents [32.49707795291693]
  Tool-integrated reasoning is difficult to scale to small language models.<n>Recently, on-policy distillation has gained popularity by supplying dense token-level supervision from a teacher.<n>We propose SOD, a step-wise on-policy distillation framework for small language model agents.
  arXiv  Detail & Related papers  (2026-05-08T13:30:42Z)
• TCOD: Exploring Temporal Curriculum in On-Policy Distillation for Multi-turn Autonomous Agents [55.27396165691312]
  We identify a key limitation of vanilla OPD in multi-turn agent settings, which we term Trajectory-Level KL Instability.<n>We propose TCOD, a framework that controls the trajectory depth exposed to the student and progressively expands it with a curriculum schedule.<n> Experimental results across four student-teacher pairs show that TCOD mitigates KL escalation and enhances KL stability throughout training, improving agent performance by up to 18 points over vanilla OPD.
  arXiv  Detail & Related papers  (2026-04-27T03:38:27Z)
• SCOPE: Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adaptive Weighting [17.504616835765617]
  We propose a dual-path adaptive training framework that routes on-policy rollouts by correctness into two complementary supervision paths.<n>SCOPE achieves an average relative improvement of 11.42% in Avg@32 and 7.30% in Pass@32 over competitive baselines.
  arXiv  Detail & Related papers  (2026-04-12T15:26:14Z)
• Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes [31.95045602299568]
  On-policy distillation (OPD) is appealing for large language model (LLM) post-training because it evaluates teacher feedback on student-generated rollouts rather than fixed teacher traces.<n>We revisit OPD from the estimator and implementation sides.<n>We identify three failure modes of sampled-token OPD: an imbalanced one-token signal, unreliable teacher guidance on student-generated prefixes, and distortions caused by tokenizer or special-token mismatch.
  arXiv  Detail & Related papers  (2026-03-26T15:35:59Z)
• Entropy-Aware On-Policy Distillation of Language Models [36.60992451188347]
  We introduce Entropy-Aware On-Policy Distillation.<n>Our key idea is augmenting the standard reverse KL objective with forward KL when teacher entropy is high.<n>It balances mode-seeking precision with mode-covering without sacrificing on-policy training efficiency.
  arXiv  Detail & Related papers  (2026-03-07T07:26:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.