Near-Policy: Accelerating On-Policy Distillation via Asynchronous Generation and Selective Packing

• URL: http://arxiv.org/abs/2605.05940v1
• Date: Thu, 07 May 2026 09:50:53 GMT
• Title: Near-Policy: Accelerating On-Policy Distillation via Asynchronous Generation and Selective Packing
• Abstract summary: Near-Policy Distillation (NPD) is an asynchronous approach that decouples student generation from training.<n>NPD achieves a 8.1x speedup over on-policy baselines and outperforms SFT by 8.09%.<n>Our method enables openPangu-Embedded-1B to reach a state-of-the-art score of 68.73%, outperforming the substantially larger Qwen3-1.7B.
• Score: 44.26853590985694
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Standard knowledge distillation for autoregressive models often suffers from distribution mismatch. While on-policy methods mitigate this by leveraging student-generated outputs, they rely on computationally expensive Reinforcement Learning (RL) frameworks. To improve efficiency, we propose Near-Policy Distillation (NPD), an asynchronous approach that decouples student generation from training. This reformulation enables Supervised Fine-Tuning (SFT) with sequence packing. However, asynchronous updates inevitably introduce policy lag and sample noise, which can cause the behavior to drift from near-policy toward off-policy. To counteract this without sacrificing efficiency, NPD integrates sparse student updates and the $Δ$-IFD filtering mechanism, a heuristic sample selection mechanism that empirically stabilizes the optimization trajectory. By filtering extreme out-of-distribution samples, $Δ$-IFD prevents noise from dominating the gradients, ensuring updates remain within a safe proximal learning zone. Empirically, the NPD framework achieves a 8.1x speedup over on-policy baselines and outperforms SFT by 8.09%. Crucially, by effectively narrowing the exploration space for subsequent RL, our method enables openPangu-Embedded-1B to reach a state-of-the-art score of 68.73%, outperforming the substantially larger Qwen3-1.7B. Codes will be released soon.

Related papers

• DiPOD: Diffusion Policy Optimization without Drifting Apart [88.2714417395158]
  RL post-training has become increasingly pivotal for improving diffusion policies.<n>Existing diffusion policy-gradient methods are often unstable and cannot achieve reliable policy improvement.<n>We propose bftextDiPOD, a diffusion policy optimization framework that maintains tight-bound behavior throughout training.
  arXiv  Detail & Related papers  (2026-06-11T18:06:04Z)
• Lagrangian Perturbation Diffusion Steering: Latent Reinforcement Learning for Generative Policies [5.170807667319543]
  We propose a lightweight adaptation method that improves a frozen generative policy by learning a compact noise-space perturbation before decoding.<n>Across RoboMimic manipulation, OpenAI Gym locomotion, and Adroit dexterous manipulation benchmarks, LP-DS improves sample efficiency, success, and return while maintaining higher action-space entropy than unconstrained noise-space steering.
  arXiv  Detail & Related papers  (2026-05-31T10:40:28Z)
• Extreme Region Policy Distillation [36.61472284280031]
  We show that aggressive multi-step optimization brings rapid initial gains, but excessive updates cause trajectory probabilities to deviate and entropy to collapse.<n>This motivates Extreme Region Policy Distillation (ERPD), a two-stage framework that decouples sample efficiency from KL efficiency.
  arXiv  Detail & Related papers  (2026-05-25T08:32:24Z)
• ANO: A Principled Approach to Robust Policy Optimization [19.004647288017814]
  We introduce a principled design space for policy optimization, demonstrating that a robust estimator must inherently suppress outliers.<n>We derive Anchored Neighborhood Optimization (ANO), a novel method that seamlessly replaces hard clipping with a redescending gradient mechanism.<n>In continuous (MuJoCo) and discrete (Atari) control, ANO establishes a robust state-of-the-art, uniquely preventing policy collapse even under highly aggressive learning rates.
  arXiv  Detail & Related papers  (2026-05-04T08:15:52Z)
• Improving Search Agent with One Line of Code [68.58667107354253]
  Tool-based Agentic Reinforcement Learning (TARL) has emerged as a promising paradigm for training search agents to interact with external tools.<n>We propose textbfSearch textbfAgent textbfPolicy textbfOptimization (textbfSAPO), which stabilizes training via a conditional token-level KL constraint.
  arXiv  Detail & Related papers  (2026-03-10T04:07:39Z)
• Coverage Improvement and Fast Convergence of On-policy Preference Learning [67.36750525893514]
  Online on-policy preference learning algorithms for language model alignment can significantly outperform their offline counterparts.<n>We analyze how the sampling policy's coverage evolves throughout on-policy training.<n>We develop principled on-policy schemes for reward distillation in the general function class setting.
  arXiv  Detail & Related papers  (2026-01-13T10:46:06Z)
• Ratio-Variance Regularized Policy Optimization for Efficient LLM Fine-tuning [48.34492357368989]
  We propose a primal-dual framework that supports stable on-policy learning and enables principled off-policy data reuse.<n>$R2VPO$ achieves superior performance with average relative gains of up to 17% over strong clipping-based baselines.
  arXiv  Detail & Related papers  (2026-01-06T14:01:42Z)
• Harnessing Bounded-Support Evolution Strategies for Policy Refinement [3.3656696418661975]
  Triangular-Distribution ES pairs bounded triangular noise with a centered-rank finite-difference estimator to deliver stable, parallelizable, gradient-free updates.<n>In a two-stage pipeline - PPO pretraining followed by TD-ES refinement - this preserves early sample efficiency while enabling robust late-stage gains.<n>Across a suite of robotic manipulation tasks, TD-ES raises success rates by 26.5% relative to PPO and greatly reduces variance, offering a simple, compute-light path to reliable refinement.
  arXiv  Detail & Related papers  (2025-11-13T03:35:52Z)
• GRPO-Guard: Mitigating Implicit Over-Optimization in Flow Matching via Regulated Clipping [63.33669214116784]
  GRPO-Guard is a simple yet effective enhancement to existing GRPO frameworks.<n>It restores a balanced and step-consistent importance ratio, ensuring that PPO clipping properly constrains harmful updates.<n>It substantially mitigates implicit over-optimization without relying on heavy KL regularization.
  arXiv  Detail & Related papers  (2025-10-25T14:51:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.