On the Position Bias of On-Policy Distillation

• URL: http://arxiv.org/abs/2606.22600v3
• Date: Fri, 26 Jun 2026 02:45:32 GMT
• Title: On the Position Bias of On-Policy Distillation
• Abstract summary: We show that Importance-Weighted On-Policy Distillation (IW-OPD) converges significantly faster than On-Policy Distillation (OPD)<n>IW-OPD achieves better final performance than standard OPD in both same-size and cross-scale settings, improving performance up to 6.9 points on AIME-2025.
• Score: 19.36735875165262
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: On-Policy Distillation (OPD) improves the learning efficiency of standard reinforcement learning through dense, token-level supervision from teachers. In the standard KL objective of OPD, token-level losses are uniformly averaged, implying equal weights for all tokens. However, we discover that not all tokens are created equal: as student rollouts grow longer, they deviate further from the teacher's distribution, leading to degraded supervision quality at later positions. As a result, OPD using only the first 30% of tokens can perform comparably to using all tokens, whereas OPD using only the last 30% of tokens barely learns anything. In this work, we provide a principled understanding of this issue through the lens of constrained optimization. Based on these insights, we derive Importance-Weighted On-Policy Distillation (IW-OPD), in which the weight assigned to each token depends on the accumulated discrepancy between the student's and teacher's distributions, naturally upweighting earlier tokens and downweighting later ones with larger deviations. We show that IW-OPD converges significantly faster than OPD, with better learning efficiency, and achieves better final performance than standard OPD in both same-size and cross-scale settings, improving performance up to 6.9 points on AIME-2025.

Related papers

• Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement [27.122890248991556]
  On-policy distillation (OPD) offers dense token-level supervision as an alternative to the sparse outcome-level advantages of reinforcement learning with verifiable rewards (RLVR)<n>We quantitatively analyze teacher supervision during OPD training and find substantial noise whose prevalence increases with teacher scale.<n>We find that learning concentrates on low log-probability tokens, and using a single fixed negative advantage matches the performance of teacher-provided ones.
  arXiv  Detail & Related papers  (2026-08-31T16:22:10Z)
• A Token-Level Analysis of Sampled-Token Reverse-KL On-Policy Distillation [59.43605629311855]
  We analyze the gradient of the per-token K2 estimator of reverse KL with respect to the student logits.<n>Our primary contribution is therefore a gradient-level characterization of reverse-KL OPD trained with the K2 estimator, with SuRe as one empirical instantiation.
  arXiv  Detail & Related papers  (2026-08-26T11:14:42Z)
• Mismatch Matters: On-Policy Distillation Beyond Token Agreement [27.396104103072844]
  We show that mismatch tokens can be mainly categorized into two types: student-excess tokens and student-deficit tokens.<n>We propose TIDE, which applies bounded Hellinger shaping to suppress the most severe sampled excesses and an analytic teacher top-$K$ injection to restore deficient probability mass.
  arXiv  Detail & Related papers  (2026-08-10T16:53:14Z)
• Distill Where You Fail: Recovering Learning Signals of Negative RL-Groups from Adaptive Teacher Guidance [50.77757355236912]
  We propose RSTG (Recovering Learning Signals via Adaptive Teacher Guidance), which applies distillation selectively and precisely where it matters most.<n> Experiments demonstrate that RSTG substantially outperforms naive GRPO+OPD by +4.02% on math and+3.05% on code.
  arXiv  Detail & Related papers  (2026-08-01T17:29:14Z)
• PriFT: Prior-Support Guided Supervised Fine-Tuning [74.65198014829393]
  Supervised fine-tuning (SFT) is an efficient approach for downstream task adaptation.<n>SFT fits fixed demonstrations token by token, including targets poorly aligned with the model's pretrained distribution.<n>We propose PriFT, which derives token weights from a frozen pretrained reference to obtain a stable reweighting signal unaffected by fine-tuning.
  arXiv  Detail & Related papers  (2026-06-08T12:14:06Z)
• Filter, Then Reweight: Rethinking Optimization Granularity in On-Policy Distillation [16.086812358369798]
  On-Policy distillation (OPD) in large language models is shifting from full-trace KL supervision toward more selective training paradigms.<n>Recent OPD methods increasingly focus on selecting which trajectories to learn from, which tokens are most informative, and which supervision signals are most reliable.<n>We propose fireicon FiRe-OPD (Filter, then Reweight), which jointly adjusts supervision signals at both trajectory and token levels.
  arXiv  Detail & Related papers  (2026-06-01T17:58:22Z)
• Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation [81.10000755917712]
  On-policy distillation transfers reasoning capabilities by training a student model on its own generated trajectories using token-level feedback from a teacher.<n>As student-generated prefixes lengthen, the teacher's next-token distribution becomes less confident and less discriminative.<n>To mitigate SFD, we introduce textbfLookahead Group Reward (ours).
  arXiv  Detail & Related papers  (2026-05-29T04:39:20Z)
• Not All Disagreement Is Learnable: Token Teachability in On-Policy Distillation [28.572361799234784]
  We show that raw KL disagreement is a coarse proxy for learning value.<n>We formalize this local compatibility as token teachability.<n>We propose Teachability-Aware OPD, a lightweight token-position selection method.
  arXiv  Detail & Related papers  (2026-05-26T10:56:46Z)
• Resolving Action Bottleneck: Agentic Reinforcement Learning Informed by Token-Level Energy [64.72195169444738]
  Policy-gradient methods treat each token in a trajectory equally, leading to uniform credit assignment.<n>We show that such uniform credit assignment largely misallocates token-level training signals.<n>We propose an embarrassingly simple token reweighting approach, ActFocus, that downweights gradients on reasoning tokens.
  arXiv  Detail & Related papers  (2026-05-14T08:33:02Z)
• Prefix Teach, Suffix Fade: Local Teachability Collapse in Strong-to-Weak On-Policy Distillation [49.117085054884676]
  On-policy distillation trains a student model on its own rollouts using dense feedback from a stronger teacher.<n>We operationalize this principle through a trajectory-specific release rule.<n> Experimental results across strong-to-weak distillation tasks indicate that this release rule consistently outperforms standard full-trajectory OPD.
  arXiv  Detail & Related papers  (2026-05-13T15:05:30Z)
• Beyond GRPO and On-Policy Distillation: An Empirical Sparse-to-Dense Reward Principle for Language-Model Post-Training [20.04756350098974]
  In settings where labeled verifiable training data is the binding constraint, each checked example should be allocated to the model and reward density where it is most informative.<n>We identify a reward-density principle that governs this allocation: sparse sequence-level reward is most useful on models that can explore and discover better behavior, while dense token-level teacher supervision is better suited for compressing that behavior into a smaller deployment model.
  arXiv  Detail & Related papers  (2026-05-12T17:57:48Z)
• Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes [31.95045602299568]
  On-policy distillation (OPD) is appealing for large language model (LLM) post-training because it evaluates teacher feedback on student-generated rollouts rather than fixed teacher traces.<n>We revisit OPD from the estimator and implementation sides.<n>We identify three failure modes of sampled-token OPD: an imbalanced one-token signal, unreliable teacher guidance on student-generated prefixes, and distortions caused by tokenizer or special-token mismatch.
  arXiv  Detail & Related papers  (2026-03-26T15:35:59Z)
• FIPO: Eliciting Deep Reasoning with Future-KL Influenced Policy Optimization [84.58281577727566]
  We present Future-KL Influenced Policy Optimization (FIPO), a reinforcement learning algorithm designed to overcome reasoning bottlenecks in large language models.<n>FIPO addresses this by incorporating discounted future-KL divergence into the policy update, creating a dense advantage formulation that re-weights tokens based on their influence on subsequent trajectory behavior.<n> evaluated on Qwen2.5-32B, FIPO extends the average chain-of-thought length from roughly 4,000 to over 10,000 tokens and increases AIME 2024 Pass@1 accuracy from 50.0% to a peak of 58.0%.
  arXiv  Detail & Related papers  (2026-03-20T10:24:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.