Eliciting Weak-to-Strong Generalization with On-Policy Reverse Distillation
Abstract Overview
This paper studies weak-to-strong generalization in on-policy distillation, asking how a stronger student model can benefit from a weaker post-trained teacher without being constrained to imitate the teacher’s final policy. The proposed method, On-Policy Reverse Distillation (OPRD), evaluates the teacher’s policy shift relative to its reference policy on student rollouts and uses that shift only to rescale the student’s verifier-driven policy gradient. By amplifying the gradient component aligned with the teacher-induced direction while leaving the orthogonal component unchanged, the method is designed to preserve the stationary points of the verifier-based optimization objective. The experiments focus on successive model transfer, multi-teacher consolidation, and a conventional strong-to-weak setting across math and reasoning benchmarks.
Novelty
The distinctive idea is to use a weak teacher’s post-training policy delta as a direction for gradient amplification rather than as a target for direct policy matching. This reframes distillation as guidance for the student’s own verifier-supported optimization, aiming to transfer useful learning signals while still allowing the student to surpass the teacher.
Results
In successive 4B-to-8B transfer, OPRD reaches weak-teacher performance with 33–67% fewer student updates than GRPO and is reported to achieve up to 22.7 percentage points better early-checkpoint performance; averaged over checkpoints, it attains 51.91 on math and 55.18 on Reasoning Gym, outperforming the listed RL and distillation baselines. In multi-teacher consolidation, it reaches teacher-level performance with 55% fewer updates than Mix-RL and achieves a 58.77 average, exceeding both the specialist-teacher average and the reported baselines. The paper also reports gains in strong-to-weak distillation, suggesting the method is not tied to a specific teacher-student capacity ordering.
Key Points
- OPRD extracts the teacher’s policy shift relative to a reference policy and uses it to amplify only the aligned component of the student’s verifier-driven gradient on student-generated rollouts.
- Unlike direct teacher matching, the method is designed to let the student continue improving beyond the weak teacher while remaining closer in response style to verifier-only RL than to teacher imitation.
- Empirical results span successive transfer, multi-teacher consolidation, and strong-to-weak distillation, with consistent improvements in update efficiency and final averaged performance over the compared baselines.
References
- arXiv: https://arxiv.org/abs/2609.08798v1
- Fugu-MT: https://fugumt.com/fugumt/paper_check/2609.08798v1
- Hugging Face Papers: https://huggingface.co/papers/2609.08798