Fast and Effective On-policy Distillation from Reasoning Prefixes

• URL: http://arxiv.org/abs/2602.15260v1
• Date: Mon, 16 Feb 2026 23:28:54 GMT
• Title: Fast and Effective On-policy Distillation from Reasoning Prefixes
• Abstract summary: On-policy distillation (OPD) samples trajectories from the student model and supervises them with a teacher at the token level.<n>We propose a simple yet effective modification of OPD: we apply the distillation objective only to prefixes of student-generated outputs and terminate each sampling early during distillation.<n>Experiments on a suite of AI-for-Math and out-of-domain benchmarks show that on-policy prefix distillation matches the performance of full OPD while reducing training FLOP by 2x-47x.
• Score: 5.0200371345178
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: On-policy distillation (OPD), which samples trajectories from the student model and supervises them with a teacher at the token level, avoids relying solely on verifiable terminal rewards and can yield better generalization than off-policy distillation. However, OPD requires expensive on-the-fly sampling of the student policy during training, which substantially increases training cost, especially for long responses. Our initial analysis shows that, during OPD, training signals are often concentrated in the prefix of each output, and that even a short teacher-generated prefix can significantly help the student produce the correct answer. Motivated by these observations, we propose a simple yet effective modification of OPD: we apply the distillation objective only to prefixes of student-generated outputs and terminate each sampling early during distillation. Experiments on a suite of AI-for-Math and out-of-domain benchmarks show that on-policy prefix distillation matches the performance of full OPD while reducing training FLOP by 2x-47x.

Related papers

• Influence-Directed Distillation: Solving the Diversity Bottleneck in Sampled-Token On-Policy Distillation [17.01350720731129]
  We propose Influence-Directed Adaptive On-Policy Distillation (IDA-OPD)<n>IDA-OPD preserves entropy-adaptive updates while replacing entropy-contracting ones with divergence-expanding advantage shrinkage.<n> Experiments on reasoning-oriented distillation show IDA-OPD consistently improves pass@$k$, inheriting the teacher's diversity through distillation.
  arXiv  Detail & Related papers  (2026-08-30T15:22:46Z)
• RL-Native Distillation: Exploiting Scored Trajectories for Few-Step Image Generation [38.215412661414604]
  Efficient text-to-image generation requires both reinforcement-learning (RL)-based reward alignment and few-step distillation.<n>We propose REST (Reward-Enhanced Scored-Trajectory Distillation), a single-stage RL-distillation co-training framework.<n>We show that REST enables few-step CFG-free inference that matches or surpasses its 40-step RL teacher.
  arXiv  Detail & Related papers  (2026-08-10T07:49:05Z)
• Simple-OPD: Demystifying Warm-up for On-policy Distillation [51.693357695326846]
  On-policy distillation (OPD) trains a student on its own rollouts with token-level supervision from teacher models.<n>We demystify warm-up for OPD from both data and training perspectives.
  arXiv  Detail & Related papers  (2026-08-07T04:47:38Z)
• Distill Where You Fail: Recovering Learning Signals of Negative RL-Groups from Adaptive Teacher Guidance [50.77757355236912]
  We propose RSTG (Recovering Learning Signals via Adaptive Teacher Guidance), which applies distillation selectively and precisely where it matters most.<n> Experiments demonstrate that RSTG substantially outperforms naive GRPO+OPD by +4.02% on math and+3.05% on code.
  arXiv  Detail & Related papers  (2026-08-01T17:29:14Z)
• Are Full Rollouts Necessary for On-Policy Distillation? [63.18243901591995]
  On-policy distillation (OPD) provides dense teacher feedback along student-generated rollouts.<n>We identify the rollout horizon as a key bottleneck in OPD that substantially impacts training efficiency.<n>We propose two simple horizon-control strategies: Progressive OPD, which gradually expands the rollout horizon during training, and Truncated OPD, which permanently performs distillation on reliable truncated rollouts.
  arXiv  Detail & Related papers  (2026-05-29T16:12:54Z)
• Scaling Reasoning Efficiently via Relaxed On-Policy Distillation [49.39199261602195]
  REOPOLD (Relaxed On-Policy Distillation) is a framework that stabilizes optimization by relaxing the strict imitation constraints of standard on-policy distillation.<n>Specifically, REOPOLD selectively and leverages rewards from the teacher through mixture-based reward clipping, entropy-based token-level dynamic sampling, and a unified exploration-to-refinement training strategy.
  arXiv  Detail & Related papers  (2026-03-11T16:26:52Z)
• OVD: On-policy Verbal Distillation [47.727229201069555]
  On-policy Verbal Distillation (OVD) is a memory-efficient framework that replaces token-level probability matching with trajectory matching.<n>OVD dramatically reduces memory consumption while enabling on-policy distillation from teacher models with verbal feedback.
  arXiv  Detail & Related papers  (2026-01-29T16:48:14Z)
• Positive-Unlabeled Reinforcement Learning Distillation for On-Premise Small Models [130.8912476550625]
  We propose a positive-unlabeled (PU) reinforcement learning distillation method for on-premise small-model deployment.<n>Our method distills the teacher's preference-optimization capability from black-box generations into a locally trainable student.<n>Experiments demonstrate that our method achieves consistently strong performance under a low-cost setting.
  arXiv  Detail & Related papers  (2026-01-28T15:14:50Z)
• Enhancing Reasoning Capabilities in SLMs with Reward Guided Dataset Distillation [0.0]
  We propose AdvDistill, a reward-guided dataset distillation framework.<n>We utilise multiple generations (responses) from a teacher for each prompt and assign rewards based on rule-based verifiers.<n>These varying and normally distributed rewards serve as weights when training student models.
  arXiv  Detail & Related papers  (2025-06-25T20:07:47Z)
• Distill Not Only Data but Also Rewards: Can Smaller Language Models Surpass Larger Ones? [58.80794196076336]
  Distilling large language models (LLMs) typically involves transferring the teacher model's responses through supervised fine-tuning (SFT)<n>We propose a novel distillation pipeline that transfers both responses and rewards.<n>Our method generates pseudo-rewards through a self-supervised mechanism that leverages the inherent structure of both teacher and student responses.
  arXiv  Detail & Related papers  (2025-02-26T20:50:11Z)
• Approximated Prompt Tuning for Vision-Language Pre-trained Models [54.326232586461614]
  In vision-language pre-trained models, prompt tuning often requires a large number of learnable tokens to bridge the gap between the pre-training and downstream tasks. We propose a novel Approximated Prompt Tuning (APT) approach towards efficient VL transfer learning.
  arXiv  Detail & Related papers  (2023-06-27T05:43:47Z)
• HomoDistil: Homotopic Task-Agnostic Distillation of Pre-trained Transformers [49.79405257763856]
  This paper focuses on task-agnostic distillation. It produces a compact pre-trained model that can be easily fine-tuned on various tasks with small computational costs and memory footprints. We propose Homotopic Distillation (HomoDistil), a novel task-agnostic distillation approach equipped with iterative pruning.
  arXiv  Detail & Related papers  (2023-02-19T17:37:24Z)
• Unbiased Knowledge Distillation for Recommendation [66.82575287129728]
  Knowledge distillation (KD) has been applied in recommender systems (RS) to reduce inference latency. Traditional solutions first train a full teacher model from the training data, and then transfer its knowledge to supervise the learning of a compact student model. We find such a standard distillation paradigm would incur serious bias issue -- popular items are more heavily recommended after the distillation.
  arXiv  Detail & Related papers  (2022-11-27T05:14:03Z)
• Efficient Scheduling of Data Augmentation for Deep Reinforcement Learning [2.837718256830632]
  In deep reinforcement learning (RL), data augmentation is widely considered as a tool to induce a set of useful priors. But even when the prior is useful for generalization, distilling it to RL agent often interferes with RL training and degenerates sample efficiency. We devise a stand-alone network distillation method to inject the consistency prior at any time (even after RL) and a simple yet efficient framework to automatically schedule the distillation.
  arXiv  Detail & Related papers  (2022-06-01T14:28:35Z)
• Robust and Resource-Efficient Data-Free Knowledge Distillation by Generative Pseudo Replay [5.3330804968579795]
  Data-Free Knowledge Distillation (KD) allows knowledge transfer from a trained neural network (teacher) to a more compact one (student) in the absence of original training data. Existing works use a validation set to monitor the accuracy of the student over real data and report the highest performance throughout the entire process. However, validation data may not be available at distillation time either, making it infeasible to record the student snapshot that achieved the peak accuracy. This is challenging because the student experiences knowledge degradation due to the distribution shift of the synthetic data. We propose to model the distribution of the previously observed synthetic samples
  arXiv  Detail & Related papers  (2022-01-09T14:14:28Z)
• Deep Semi-supervised Knowledge Distillation for Overlapping Cervical Cell Instance Segmentation [54.49894381464853]
  We propose to leverage both labeled and unlabeled data for instance segmentation with improved accuracy by knowledge distillation. We propose a novel Mask-guided Mean Teacher framework with Perturbation-sensitive Sample Mining. Experiments show that the proposed method improves the performance significantly compared with the supervised method learned from labeled data only.
  arXiv  Detail & Related papers  (2020-07-21T13:27:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.