2026-07-30 Daily Report: $β$-OPSD: Deriving with Policy Optimization, Training with Self-Distillation

$β$-OPSD: Deriving with Policy Optimization, Training with Self-Distillation

Authors Jiawei Xu, Minghui Liu, Juzheng Zhang, Tom Goldstein, Furong Huang

Affiliations University of Maryland

Categories Method / Self-Distillation / Online self-distillation methods, Method / Policy Optimization / Policy optimization derivation techniques, Evaluation / Model Improvement / Improving inference language models

License CC BY 4.0

Abstract Overview

This paper argues that vanilla on-policy self-distillation (OPSD) can be viewed as the β=1 case of a broader KL-regularized policy-optimization objective that balances movement toward a privileged teacher against staying close to a reference policy. From this view, the authors derive an optimal policy that lies on a geometric interpolation path between the reference policy and the teacher. Instead of optimizing the resulting reinforcement-learning objective directly, they construct a practical distillation target by interpolating token-level logits along a scheduled teacher weight. They also add return-to-go credit assignment so token updates better reflect sequence-level mismatch. The method is evaluated on mathematical reasoning benchmarks with Qwen3 models from 1.7B to 8B parameters.

Novelty

The main novelty is the re-derivation of OPSD as a special case of KL-regularized policy optimization, which exposes β as an explicit regularization parameter rather than an implicit fixed choice. This leads to β-OPSD, where the RL-derived optimal target is implemented efficiently through scheduled logit interpolation and paired with return-to-go credit assignment for on-policy distillation.

Results

On AIME 2024, AIME 2025, and HMMT 2025, β-OPSD improves the average score over vanilla OPSD for Qwen3-1.7B, 4B, and 8B by 5.74, 1.76, and 1.66 points, respectively, and achieves the best overall average among the compared methods in Table 1. The gains are largest for Qwen3-1.7B, where the method improves over vanilla OPSD by 9.16 points on AIME 2024, 5.27 on AIME 2025, and 2.78 on HMMT 2025. Ablations further show that the interpolant target outperforms direct teacher distillation under the same return-to-go setup, and that return-to-go credit assignment outperforms a local token-gradient variant.

Key Points

Vanilla OPSD is reformulated as the β=1 member of a KL-regularized policy-optimization family, making the strength of reference anchoring explicit and tunable.
The practical algorithm replaces direct teacher matching with a scheduled logit interpolant between a reference/student policy and a privileged teacher, and uses return-to-go weighting for sequence-aware credit assignment.
Experiments on competition-level math reasoning benchmarks show consistent average improvements over vanilla OPSD across model scales, with ablations supporting both the interpolant target and the return-to-go estimator.

References

arXiv: https://arxiv.org/abs/2607.28582v1
Fugu-MT: https://fugumt.com/fugumt/paper_check/2607.28582v1
Hugging Face Papers: https://huggingface.co/papers/2607.28582
Project: https://furong-huang.com

Project