Data-free On-policy Distillation

• URL: http://arxiv.org/abs/2609.14193v1
• Date: Sat, 12 Sep 2026 23:51:35 GMT
• Title: Data-free On-policy Distillation
• Abstract summary: On-policy distillation (OPD) has become a standard component of frontier post-training pipelines.<n>But how much its training data actually contributes has gone largely unexamined.<n>We take this to its limit with Data-free On-policy Distillation (DF-OPD)<n>DF-OPD matches and even surpasses real data, and the questions it produces track the teacher's own post-training data.
• Score: 49.537587070291444
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: On-policy distillation (OPD) has become a standard component of frontier post-training pipelines, yet how much its training data actually contributes has gone largely unexamined. On the two teacher-student pairings most common in practice, we find OPD almost indifferent to its data: eight prompts already match a 17k-problem dataset, and three independently built datasets whose difficulty and teacher-student KL differ several-fold produce nearly indistinguishable training curves. Two causes account for this. First, the unit of data in OPD is the state a prompt leads to, not the prompt itself: a single prompt keeps exposing new teacher correction as sampling continues, while the marginal value of additional prompts collapses after eight. Second, replacing mathematics with competitive programming still recovers over ninety percent of the in-domain gain, indicating that OPD transfers the teacher's mode of reasoning rather than knowledge related to the data. We take this to its limit with Data-free On-policy Distillation (DF-OPD), in which the teacher writes its own training questions under a simple prompt -- no external data, no filtering -- leaving a system of just two policies. DF-OPD matches and even surpasses real data, and the questions it produces track the teacher's own post-training data on three key diagnostics of training dynamics, which other real datasets do not. Applied to multi-teacher distillation, where the (prompt, domain) pairs normally have to be derived from post-training data that is often out of reach, 1k self-generated questions close 98.5% of the available headroom, even surpassing the 96.6% reached with 7k real examples. Moreover, together these results invite a reassessment of the role data plays in OPD.

Related papers

• What Matters in On-Policy Distillation? A Perspective on Data Efficiency and Data Selection [25.414327410290543]
  On-Policy Distillation (OPD) has emerged as a widely adopted post-training paradigm for enhancing large language models in reasoning domains.<n>This paper presents a empirical study of data efficiency and data selection in OPD.
  arXiv  Detail & Related papers  (2026-09-04T14:32:08Z)
• Rethinking On-Policy Distillation of Large Language Models II: One Training Example [31.843793429278808]
  On-policy distillation combines student-generated rollouts with dense token-level supervision from a teacher.<n>One-shot OPD keeps improving for hundreds of steps and recovers most of full-data OPD's gain across task domains and model families.<n>We explain this result through the states visited during training and the rate at which the student aligns with the teacher.
  arXiv  Detail & Related papers  (2026-09-03T17:54:38Z)
• OPOD: On-Policy Omni Distillation [55.440058537827134]
  Training a single model on pooled multimodal data often fails to match models specialized for individual modalities.<n>We present On-Policy Omni Distillation (OPD), which routes each student response to the matching text, image, or audio teacher.<n>OPD achieves the best average score at every scale, reaching 70.8, 51.7, and 46.2.
  arXiv  Detail & Related papers  (2026-07-23T04:55:04Z)
• Self-Verified Distillation: Your Language Model Is Secretly Its Own Synthetic Data Pipeline [56.53954182896384]
  We propose Self-Verified Distillation, a simple post-training refinement algorithm for large language models.<n>Self-Verified Distillation generates candidate solutions to unlabeled seed questions.<n>It filters them using prompt-based self-verification, and trains on the resulting self-curated dataset.<n>We find that sampling more candidate generations and using a larger verification budget during training data construction produces higher-quality self-curated data.
  arXiv  Detail & Related papers  (2026-05-20T17:26:10Z)
• DeltaPrompts: Escaping the Zero-Delta Trap in Multimodal Distillation [49.98710755440242]
  Distillation enables compact Vision-Language Models (VLMs) to obtain strong reasoning capabilities.<n>We reveal a critical inefficiency in this approach: up to 69% of the prompts in standard chart / document reasoning datasets are effectively zero-delta.<n>We propose a staged synthesis pipeline that repurposes existing datasets as seeds, actively targeting student failure modes to produce better prompts.
  arXiv  Detail & Related papers  (2026-05-15T02:04:12Z)
• Offline Reinforcement Learning: Role of State Aggregation and Trajectory Data [64.93285984013833]
  We revisit the problem of offline reinforcement learning with value function realizability but without Bellman. Previous work by Xie and Jiang (2021) and Foster et al. (2022) left open the question whether a concentrability coefficient along with trajectory-based offline data unifies a bounded sample complexity.
  arXiv  Detail & Related papers  (2024-03-25T18:28:45Z)
• Sampling to Distill: Knowledge Transfer from Open-World Data [28.74835717488114]
  We propose a novel Open-world Data Sampling Distillation (ODSD) method for the Data-Free Knowledge Distillation (DFKD) task without the redundant generation process. First, we try to sample open-world data close to the original data's distribution by an adaptive sampling module. Then, we build structured relationships of multiple data examples to exploit data knowledge through the student model itself and the teacher's structured representation.
  arXiv  Detail & Related papers  (2023-07-31T12:05:55Z)
• Decentralized Federated Learning Preserves Model and Data Privacy [77.454688257702]
  We propose a fully decentralized approach, which allows to share knowledge between trained models. Students are trained on the output of their teachers via synthetically generated input data. The results show that an untrained student model, trained on the teachers output reaches comparable F1-scores as the teacher.
  arXiv  Detail & Related papers  (2021-02-01T14:38:54Z)
• Towards Zero-Shot Knowledge Distillation for Natural Language Processing [9.223848704267088]
  Knowledge Distillation (KD) is a common algorithm used for model compression across a variety of deep learning based natural language processing (NLP) solutions. In its regular manifestations, KD requires access to the teacher's training data for knowledge transfer to the student network. We present to the best of our knowledge, the first work on Zero-Shot Knowledge Distillation for NLP, where the student learns from the much larger teacher without any task specific data.
  arXiv  Detail & Related papers  (2020-12-31T08:16:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.