Related papers: Less is More: Early Stopping Rollout for On-Policy Distillation

Less is More: Early Stopping Rollout for On-Policy Distillation

URL: http://arxiv.org/abs/2605.27028v1
Date: Tue, 26 May 2026 13:49:37 GMT
Title: Less is More: Early Stopping Rollout for On-Policy Distillation
Abstract summary: Early Stopping Rollout (ESR) is a simple yet effective distillation strategy that simply restricts the rollout generation to the first response tokens.<n>We show that ESR both surpasses the full rollout OPD performance across model size, family, tasks and training regime.
Score: 39.392596318514244
License: http://creativecommons.org/licenses/by/4.0/
Abstract: On-policy distillation has recently emerged as a promising alternative to standard sequence-level imitation, training a student by scoring its own rollouts with a teacher model. However, we observe ``Off-policy Teacher Decay'' problem in this paradigm: for the later tokens, with student's earlier trajectory as context that is off-policy to the teacher, the teacher's ability to produce a corrective score would decay, and may fall back to token-completion behavior learned in the pre-training stage. We empirically verify this problem, and we propose Early Stopping Rollout (ESR) to fix it: a simple yet effective distillation strategy that simply restricts the rollout generation to the first response tokens. We show that ESR both surpasses the full rollout OPD performance across model size, family, tasks and training regime, and exhibit much higher GPU efficiency and training stability, especially under cross model family scenarios. We further investigate the mechanism behind this surprising performance and discovered "Cascading Alignment" and "Sub-mode Commitment" effect of ESR that may explain why it works effectively and even sometimes exceeding the teacher model performance. Besides, we show that this position-based token selection strategy cannot be fully explainable by KL divergence and entropy signals.

Related papers

HERO: Hindsight-Enhanced Reflection from Environment Observations for Agentic Self-Distillation [50.53459634301361]
HERO is a hindsight-enhanced self-distillation framework that uses next environment observations as locally aligned feedback.<n> HERO improves task success and reduces unnecessary turns over environment-feedback-only self-distillation and GRPO.
arXiv Detail & Related papers (2026-06-10T01:35:34Z)
EDGE-OPD: Internalizing Privileged Context with Evidence Guided On-Policy Distillation [5.310892696470208]
On-Policy Distillation (OPD) has gained wide attraction as an LLM post-training paradigm.<n>The challenge in this approach is that the privileged information can change model behavior more than intended.<n>We propose EviDence GuidEd On-Policy Distillation (EDGE-OPD), a modification of OPD with two distinct characteristics.
arXiv Detail & Related papers (2026-05-22T10:55:15Z)
Prefix Teach, Suffix Fade: Local Teachability Collapse in Strong-to-Weak On-Policy Distillation [49.117085054884676]
On-policy distillation trains a student model on its own rollouts using dense feedback from a stronger teacher.<n>We operationalize this principle through a trajectory-specific release rule.<n> Experimental results across strong-to-weak distillation tasks indicate that this release rule consistently outperforms standard full-trajectory OPD.
arXiv Detail & Related papers (2026-05-13T15:05:30Z)
Prune-OPD: Efficient and Reliable On-Policy Distillation for Long-Horizon Reasoning [66.52232008796294]
Prune-OPD dynamically aligns training budgets with supervision quality.<n>It reduces training time by 37.6%--68.0% while preserving, and often improving, performance on challenging benchmarks.
arXiv Detail & Related papers (2026-05-08T14:38:53Z)
Preference-Based Self-Distillation: Beyond KL Matching via Reward Regularization [18.027254451537342]
Existing self-distillation methods largely reduce learning to KL matching toward the context-augmented teacher model.<n>We propose textbfPreference-textbfBased textbfSelf-textbfDistillation (textbfPBSD), which revisits on-policy self-distillation through a reward-regularized perspective.
arXiv Detail & Related papers (2026-05-06T15:31:50Z)
TCOD: Exploring Temporal Curriculum in On-Policy Distillation for Multi-turn Autonomous Agents [55.27396165691312]
We identify a key limitation of vanilla OPD in multi-turn agent settings, which we term Trajectory-Level KL Instability.<n>We propose TCOD, a framework that controls the trajectory depth exposed to the student and progressively expands it with a curriculum schedule.<n> Experimental results across four student-teacher pairs show that TCOD mitigates KL escalation and enhances KL stability throughout training, improving agent performance by up to 18 points over vanilla OPD.
arXiv Detail & Related papers (2026-04-27T03:38:27Z)
Hindsight-Anchored Policy Optimization: Turning Failure into Feedback in Sparse Reward Settings [7.554296968064274]
We present Hindsight-Anchored Policy optimization (HAPO) for Reinforcement Learning with Verifiable Rewards (RLVR)<n>HAPO employs a hindsight mechanism that selectively anchors optimization to teacher demonstrations during failure.<n>We show that HAPO achieves textitasymptotic consistency: by naturally annealing the teacher signal as the policy improves, HAPO recovers the unbiased on-policy gradient.
arXiv Detail & Related papers (2026-03-11T21:33:41Z)
DDPG++: Striving for Simplicity in Continuous-control Off-Policy Reinforcement Learning [95.60782037764928]
We show that simple Deterministic Policy Gradient works remarkably well as long as the overestimation bias is controlled. Second, we pinpoint training instabilities, typical of off-policy algorithms, to the greedy policy update step. Third, we show that ideas in the propensity estimation literature can be used to importance-sample transitions from replay buffer and update policy to prevent deterioration of performance.
arXiv Detail & Related papers (2020-06-26T20:21:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.