Scaling Reasoning Efficiently via Relaxed On-Policy Distillation

• URL: http://arxiv.org/abs/2603.11137v1
• Date: Wed, 11 Mar 2026 16:26:52 GMT
• Title: Scaling Reasoning Efficiently via Relaxed On-Policy Distillation
• Abstract summary: REOPOLD (Relaxed On-Policy Distillation) is a framework that stabilizes optimization by relaxing the strict imitation constraints of standard on-policy distillation.<n>Specifically, REOPOLD selectively and leverages rewards from the teacher through mixture-based reward clipping, entropy-based token-level dynamic sampling, and a unified exploration-to-refinement training strategy.
• Score: 49.39199261602195
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: On-policy distillation is pivotal for transferring reasoning capabilities to capacity-constrained models, yet remains prone to instability and negative transfer. We show that on-policy distillation can be interpreted, both theoretically and empirically, as a form of policy optimization, where the teacher-student log-likelihood ratio acts as a token reward. From this insight, we introduce REOPOLD (Relaxed On-Policy Distillation) a framework that stabilizes optimization by relaxing the strict imitation constraints of standard on-policy distillation. Specifically, REOPOLD temperately and selectively leverages rewards from the teacher through mixture-based reward clipping, entropy-based token-level dynamic sampling, and a unified exploration-to-refinement training strategy. Empirically, REOPOLD surpasses its baselines with superior sample efficiency during training and enhanced test-time scaling at inference, across mathematical, visual, and agentic tool-use reasoning tasks. Specifically, REOPOLD outperforms recent RL approaches achieving 6.7~12x greater sample efficiency and enables a 7B student to match a 32B teacher in visual reasoning with a ~3.32x inference speedup.

Related papers

• Eliciting Weak-to-Strong Generalization with On-Policy Reverse Distillation [41.854144563456046]
  Weak-to-strong generalization asks whether stronger models can learn from weaker supervisors and surpass them.<n>We introduce On-Policy Reverse Distillation, which evaluates the teacher's policy shift and amplifies the student's verifier-driven policy gradient.<n>In both successive model transfer and multi-teacher distillation, OPRD achieves prohibitively higher performance with fewer student updates than existing RL and distillation approaches.
  arXiv  Detail & Related papers  (2026-09-08T14:26:35Z)
• Post-Training at the Edge of Detectability: A Game-Theoretic Approach to Fine-Tuning [58.088509347353465]
  Reinforcement learning (RL) fine-tuning is widely used in language model training.<n>We propose a game-theoretic framework that gives this trade-off an explicit statistical interpretation.
  arXiv  Detail & Related papers  (2026-07-29T00:19:09Z)
• On-Policy Self-Distillation with Sampled Demonstrations Reduces Output Diversity [5.35852583397955]
  On-policy self-distillation achieves strong pass@1 accuracy by using a single model as both teacher and student.<n>We show that this could come at a hidden cost: rollout diversity decreases and pass@k curves flatten.
  arXiv  Detail & Related papers  (2026-06-24T17:59:02Z)
• Preference-Based Self-Distillation: Beyond KL Matching via Reward Regularization [18.027254451537342]
  Existing self-distillation methods largely reduce learning to KL matching toward the context-augmented teacher model.<n>We propose textbfPreference-textbfBased textbfSelf-textbfDistillation (textbfPBSD), which revisits on-policy self-distillation through a reward-regularized perspective.
  arXiv  Detail & Related papers  (2026-05-06T15:31:50Z)
• Recovering Hidden Reward in Diffusion-Based Policies [20.31180123599115]
  EnergyFlow is a framework that unifies generative action modeling with inverse reinforcement learning.<n>Under maximum-entropy optimality, the score function learned via denoising score matching recovers the gradient of the expert's soft Q-function.<n>We show that constraining the learned field to be conservative reduces hypothesis complexity and tightens out-of-distribution generalization bounds.
  arXiv  Detail & Related papers  (2026-05-01T12:52:56Z)
• Fast and Effective On-policy Distillation from Reasoning Prefixes [5.0200371345178]
  On-policy distillation (OPD) samples trajectories from the student model and supervises them with a teacher at the token level.<n>We propose a simple yet effective modification of OPD: we apply the distillation objective only to prefixes of student-generated outputs and terminate each sampling early during distillation.<n>Experiments on a suite of AI-for-Math and out-of-domain benchmarks show that on-policy prefix distillation matches the performance of full OPD while reducing training FLOP by 2x-47x.
  arXiv  Detail & Related papers  (2026-02-16T23:28:54Z)
• KEPO: Knowledge-Enhanced Preference Optimization for Reinforcement Learning with Reasoning [24.072603982041798]
  Reinforcement learning has emerged as a promising paradigm for inducing explicit reasoning behaviors in large language and vision-language models.<n>However, reasoning-oriented RL post-training remains fundamentally challenging due to sparse trajectory-level rewards.<n>Recent on-policy distillation methods introduce dense teacher supervision to stabilize optimization, but apply it uniformly across all generated trajectories.
  arXiv  Detail & Related papers  (2026-01-30T23:28:37Z)
• A Step Back: Prefix Importance Ratio Stabilizes Policy Optimization [58.116300485427764]
  Reinforcement learning post-training can elicit reasoning behaviors in large language models.<n> token-level correction often leads to unstable training dynamics when the degree of off-policyness is large.<n>We propose a simple yet effective objective, Minimum Prefix Ratio (MinPRO)
  arXiv  Detail & Related papers  (2026-01-30T08:47:19Z)
• Stabilizing Reinforcement Learning with LLMs: Formulation and Practices [61.361819972410046]
  We show why and under what conditions the true sequence-level reward can be optimized via a surrogate token-level objective in policy gradient methods such as REINFORCE.<n>This insight provides a principled explanation for the crucial role of several widely adopted techniques in stabilizing RL training.
  arXiv  Detail & Related papers  (2025-12-01T07:45:39Z)
• CurES: From Gradient Analysis to Efficient Curriculum Learning for Reasoning LLMs [53.749193998004166]
  Curriculum learning plays a crucial role in enhancing the training efficiency of large language models.<n>We propose CurES, an efficient training method that accelerates convergence and employs Bayesian posterior estimation to minimize computational overhead.
  arXiv  Detail & Related papers  (2025-10-01T15:41:27Z)
• Efficient Preference-based Reinforcement Learning via Aligned Experience Estimation [37.36913210031282]
  Preference-based reinforcement learning (PbRL) has shown impressive capabilities in training agents without reward engineering. We propose SEER, an efficient PbRL method that integrates label smoothing and policy regularization techniques.
  arXiv  Detail & Related papers  (2024-05-29T01:49:20Z)
• On the Reuse Bias in Off-Policy Reinforcement Learning [28.29153543457396]
  Reuse Bias is the bias in off-policy evaluation caused by the reuse of the replay buffer for evaluation and optimization. We show that the off-policy evaluation and optimization of the current policy with the data from the replay buffer result in an overestimation of the objective. We present a novel Bias-Regularized Importance Sampling (BIRIS) framework along with practical algorithms, which can alleviate the negative impact of the Reuse Bias.
  arXiv  Detail & Related papers  (2022-09-15T06:20:36Z)
• Off-policy Reinforcement Learning with Optimistic Exploration and Distribution Correction [73.77593805292194]
  We train a separate exploration policy to maximize an approximate upper confidence bound of the critics in an off-policy actor-critic framework. To mitigate the off-policy-ness, we adapt the recently introduced DICE framework to learn a distribution correction ratio for off-policy actor-critic training.
  arXiv  Detail & Related papers  (2021-10-22T22:07:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.