On-Policy Delta Distillation

• URL: http://arxiv.org/abs/2607.15161v1
• Date: Thu, 16 Jul 2026 16:07:19 GMT
• Title: On-Policy Delta Distillation
• Abstract summary: On-policy distillation is an alternative post-training method in reinforcement learning.<n>We introduce a new distillation reward, termed the delta signal, instead of directly imitating the teacher's output distribution.<n>We show that the delta signal substantially improves on-policy distillation and refer to the new distillation method as On-Policy Delta Distillation (OPD$2$)
• Score: 61.76889440384448
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: On-policy distillation is an alternative post-training method in reinforcement learning that alleviates the constraints imposed by reward models by providing token-level supervision from a teacher model. Although on-policy distillation has been studied and applied across various settings, its fundamental design remains underexplored. In this paper, we introduce a new distillation reward, termed the delta signal, instead of directly imitating the teacher's output distribution. The delta signal is defined as the difference between the teacher model and its base model prior to instruction tuning for reasoning capability. It therefore captures the changes induced by reasoning tuning and provides a more direct signal for transferring reasoning capabilities. Using extensive empirical evidence, we show that the delta signal substantially improves on-policy distillation and refer to the new distillation method as On-Policy Delta Distillation (OPD$^2$). Experiments across mathematics, science, and code-reasoning benchmarks demonstrate that OPD$^2$ consistently outperforms conventional on-policy distillation, enabling reasoning LLMs to achieve strong performance with only a short post-training period. Code will be available at https://github.com/naver-ai/opd2

Related papers

• Reference-Based Distillation Detection in LLMs [17.877444310396758]
  We introduce a distillation detection method based on reference-based membership inference.<n>We develop a hybrid evaluation spanning both controlled distillation experiments and real-world models.<n>Applying our method to contemporary models yields new evidence regarding potential distillation relationships involving QwQ, DeepSeek-R1, and GPT-OSS.
  arXiv  Detail & Related papers  (2026-06-19T06:59:19Z)
• Adversarial Dual On-Policy Distillation from Expressive Teacher [53.63143943452487]
  Flow Matching teacher is learned from demonstrations and co-trained with a lightweight student.<n>The teacher provides two complementary signals on student rollouts.<n>The action channel supplies dense local targets at student-visited states, stabilizing exploitation.
  arXiv  Detail & Related papers  (2026-05-26T14:38:03Z)
• DeltaPrompts: Escaping the Zero-Delta Trap in Multimodal Distillation [49.98710755440242]
  Distillation enables compact Vision-Language Models (VLMs) to obtain strong reasoning capabilities.<n>We reveal a critical inefficiency in this approach: up to 69% of the prompts in standard chart / document reasoning datasets are effectively zero-delta.<n>We propose a staged synthesis pipeline that repurposes existing datasets as seeds, actively targeting student failure modes to produce better prompts.
  arXiv  Detail & Related papers  (2026-05-15T02:04:12Z)
• Validity-Calibrated Reasoning Distillation [4.277471273303775]
  Reasoning distillation aims to transfer multi-step reasoning capabilities from large language models to smaller, more efficient ones.<n>Recent methods rely on static teacher-student hierarchies and frame distillation as trajectory imitation.<n>We propose validity-calibrated reasoning distillation, a framework that treats reasoning distillation as a problem of local learning-signal allocation.
  arXiv  Detail & Related papers  (2026-04-14T12:32:12Z)
• PACED: Distillation and Self-Distillation at the Frontier of Student Competence [17.384089089363382]
  Paced is a framework that concentrates distillation on the zone of proximal development.<n>On instruction-tuned models with reverse KL, gains are exceeding baselines.<n>A forward-KL-then-reverse-KL schedule yields the strongest results in our setting.
  arXiv  Detail & Related papers  (2026-03-11T18:00:05Z)
• Towards Training One-Step Diffusion Models Without Distillation [72.80423908458772]
  We introduce a family of new training methods that entirely forgo teacher score supervision.<n>We find that initializing the student model with the teacher's weights remains critical.
  arXiv  Detail & Related papers  (2025-02-11T23:02:14Z)
• Knowledge Distillation with Refined Logits [31.205248790623703]
  We introduce Refined Logit Distillation (RLD) to address the limitations of current logit distillation methods.<n>Our approach is motivated by the observation that even high-performing teacher models can make incorrect predictions.<n>Our method can effectively eliminate misleading information from the teacher while preserving crucial class correlations.
  arXiv  Detail & Related papers  (2024-08-14T17:59:32Z)
• Unbiased Knowledge Distillation for Recommendation [66.82575287129728]
  Knowledge distillation (KD) has been applied in recommender systems (RS) to reduce inference latency. Traditional solutions first train a full teacher model from the training data, and then transfer its knowledge to supervise the learning of a compact student model. We find such a standard distillation paradigm would incur serious bias issue -- popular items are more heavily recommended after the distillation.
  arXiv  Detail & Related papers  (2022-11-27T05:14:03Z)
• Knowledge distillation via adaptive instance normalization [52.91164959767517]
  We propose a new knowledge distillation method based on transferring feature statistics from the teacher to the student. Our method goes beyond the standard way of enforcing the mean and variance of the student to be similar to those of the teacher. We show that our distillation method outperforms other state-of-the-art distillation methods over a large set of experimental settings.
  arXiv  Detail & Related papers  (2020-03-09T17:50:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.