Rethinking On-Policy Distillation of Large Language Models II: One Training Example

• URL: http://arxiv.org/abs/2609.04172v1
• Date: Thu, 03 Sep 2026 17:54:38 GMT
• Title: Rethinking On-Policy Distillation of Large Language Models II: One Training Example
• Abstract summary: On-policy distillation combines student-generated rollouts with dense token-level supervision from a teacher.<n>One-shot OPD keeps improving for hundreds of steps and recovers most of full-data OPD's gain across task domains and model families.<n>We explain this result through the states visited during training and the rate at which the student aligns with the teacher.
• Score: 31.843793429278808
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: On-policy distillation (OPD) combines student-generated rollouts with dense token-level supervision from a teacher. Existing work has mainly studied its algorithmic behavior, leaving the role of training data unclear. We examine this role at the data-minimal limit by training on a single query. One-shot OPD keeps improving for hundreds of steps and recovers most of full-data OPD's gain across task domains and model families. We explain this result through the states visited during training and the rate at which the student aligns with the teacher. We measure \emph{state coverage}, the fraction of the states full-data OPD visits that a query set's rollouts reach. A single query already reaches \(71.5\%\), most of it within the first 100 steps. Adding semantically distinct queries raises coverage and validation accuracy together, until 16 queries reach \(98.9\%\) and match full-data training. Yet alignment slows at a similar pace whether OPD trains on one query or the whole dataset, and even a fixed set of states takes hundreds of steps to absorb. OPD is therefore data-overfed but algorithm-starved. Its rollouts quickly expose broad supervision, while the student absorbs that supervision increasingly slowly. The state-coverage result extends to multi-teacher OPD, where 16 semantically diverse queries per domain match full-data MOPD. As a further stress test, content-light templates and off-domain WildChat queries also approach the real-query baseline. Task content and induced state coverage can therefore come apart. We hope these findings direct future work toward the step efficiency of OPD, and prompt a re-examination of the data and the mechanisms behind its recent successes in frontier post-training.

Related papers

• Data-free On-policy Distillation [49.537587070291444]
  On-policy distillation (OPD) has become a standard component of frontier post-training pipelines.<n>But how much its training data actually contributes has gone largely unexamined.<n>We take this to its limit with Data-free On-policy Distillation (DF-OPD)<n>DF-OPD matches and even surpasses real data, and the questions it produces track the teacher's own post-training data.
  arXiv  Detail & Related papers  (2026-09-12T23:51:35Z)
• MemOPD: On-Policy Distillation through Memory State Alignment for Long-Horizon Agents [48.05798760668219]
  Long-horizon agents accumulate growing contexts during interaction, impairing performance and stability.<n>We propose Memory-Aligned On-Policy Distillation (MemOPD) to mitigate this problem.<n>MemOPD records the inputs and sampled outputs of each model invocation, restores its original token positions, and packs the reconstructed invocations for efficient teacher scoring.
  arXiv  Detail & Related papers  (2026-08-07T10:18:31Z)
• Multi-Turn On-Policy Distillation with Prefix Replay [73.10000453999088]
  We propose Replayed-Prefix On-Policy Distillation (ReOPD) for agentic tasks.<n>ReOPD reuses pre-collected teacher trajectories as replayed prefixes.<n>It preserves or improves OPD-level accuracy, uses zero tool calls during student training, and is at least 4$times$ faster per training step than OPD.
  arXiv  Detail & Related papers  (2026-07-06T07:56:53Z)
• DemoPSD: Disagreement-Modulated Policy Self-Distillation [49.30809363804017]
  On-policy self-distillation has emerged as a practical method for training large language models.<n>DemoPSD steers the student toward a weighted geometric combination of the teacher and student distributions.
  arXiv  Detail & Related papers  (2026-07-02T17:58:29Z)
• TCOD: Exploring Temporal Curriculum in On-Policy Distillation for Multi-turn Autonomous Agents [55.27396165691312]
  We identify a key limitation of vanilla OPD in multi-turn agent settings, which we term Trajectory-Level KL Instability.<n>We propose TCOD, a framework that controls the trajectory depth exposed to the student and progressively expands it with a curriculum schedule.<n> Experimental results across four student-teacher pairs show that TCOD mitigates KL escalation and enhances KL stability throughout training, improving agent performance by up to 18 points over vanilla OPD.
  arXiv  Detail & Related papers  (2026-04-27T03:38:27Z)
• Transformer-Driven Modeling of Variable Frequency Features for Classifying Student Engagement in Online Learning [2.127312905562737]
  This paper proposes EngageFormer, a transformer based architecture with sequence pooling using video modality for engagement classification.<n>The proposed architecture computes three views from the input video and processes them in parallel using transformer encoders.<n>A learning centered affective state dataset is curated from existing open source databases.
  arXiv  Detail & Related papers  (2025-02-15T14:37:09Z)
• Adaptive Retention & Correction: Test-Time Training for Continual Learning [114.5656325514408]
  A common problem in continual learning is the classification layer's bias towards the most recent task.<n>We name our approach Adaptive Retention & Correction (ARC)<n>ARC achieves an average performance increase of 2.7% and 2.6% on the CIFAR-100 and Imagenet-R datasets.
  arXiv  Detail & Related papers  (2024-05-23T08:43:09Z)
• QUERT: Continual Pre-training of Language Model for Query Understanding in Travel Domain Search [15.026682829320261]
  We propose QUERT, A Continual Pre-trained Language Model for QUERy Understanding in Travel Domain Search. Quert is jointly trained on four tailored pre-training tasks to the characteristics of query in travel domain search. To check on the improvement of QUERT to online business, we deploy QUERT and perform A/B testing on Fliggy APP.
  arXiv  Detail & Related papers  (2023-06-11T15:39:59Z)
• Data-Efficient Ranking Distillation for Image Retrieval [15.88955427198763]
  Recent approaches tackle this issue using knowledge distillation to transfer knowledge from a deeper and heavier architecture to a much smaller network. In this paper we address knowledge distillation for metric learning problems. Unlike previous approaches, our proposed method jointly addresses the following constraints i) limited queries to teacher model, ii) black box teacher model with access to the final output representation, andiii) small fraction of original training data without any ground-truth labels.
  arXiv  Detail & Related papers  (2020-07-10T10:59:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.