Latent On-Policy Self-Distillation

• URL: http://arxiv.org/abs/2608.13040v1
• Date: Thu, 13 Aug 2026 10:05:51 GMT
• Title: Latent On-Policy Self-Distillation
• Abstract summary: On-policy self-distillation (OPSD) offers an effective pathway by using a privileged self-teacher to provide dense supervision on the student's own trajectories.<n>Existing methods still rely heavily on designer-specified privileged artifacts.<n>We introduce Latent On-Policy Self-Distillation (LOPD), which makes the teacher's privileged context itself learnable end-to-end from experience.
• Score: 55.57800223145407
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Enabling agents to learn from experience and internalize it into their policy has become a central problem in self-evolving AI. On-policy self-distillation (OPSD) offers an effective pathway by using a privileged self-teacher to provide dense supervision on the student's own trajectories; however, existing methods still rely heavily on designer-specified privileged artifacts (e.g., answers, feedback, skills, or trajectories), limiting the end-to-end learnability and scalability required for continual self-improvement. In this work, we introduce Latent On-Policy Self-Distillation (LOPD), which, rather than proposing another hand-crafted OPSD variant with a newly prescribed form of privileged context, makes the teacher's privileged context itself learnable end-to-end from experience. Technically, LOPD retrieves relevant experiences and composes them into continuous latent tokens that condition a self-teacher, while the student generates trajectories from the task and interaction history and receives dense token-level supervision at every visited prefix. We further introduce a privileged-margin objective to stabilize and regulate the learning of latent context. Empirically, LOPD demonstrates (I) strong performance, outperforming RLVR and representative OPSD methods including OPSD, SDPO, and Skill-SD across both agentic tool use and code generation; and (II) high learning efficiency, surpassing GRPO and Skill-SD with less than 30% of their rollout budget. Ablation studies further provide direct evidence that making privileged context learnable is necessary for realizing these gains. Together, these results position LOPD as a step toward a more scalable and self-directed paradigm for agent evolution.

Related papers

• Bidirectional Context Self-Distillation for Reinforcement Learning of Skill-Based LLM Agents [11.967142343676171]
  We propose BCSD (Bi Context Self-Distillation), a framework that combines self-distillation and reinforcement learning to train agents to use external skills more effectively.<n> Experiments on ALFWorld and WebShop demonstrate that BCSD achieves the strongest overall performance across model scales.
  arXiv  Detail & Related papers  (2026-08-10T12:53:06Z)
• From Scoring to Acting: Outcome-Verified Comparative Self-Distillation for LLM Agents [16.67357658768852]
  We propose Outcome-Verified Comparative Self-Distillation (OVCSD)<n>OVCSD organizes failed student rollouts into a prefix tree, adaptively invokes a skill-conditioned teacher from student-reached states, and retains only outcome-verified successful continuations.<n> Experiments on ALFWorld and WebShop across three model scales show that OVCSD consistently outperforms skill-free RL and existing self-distillation baselines.
  arXiv  Detail & Related papers  (2026-07-30T09:47:58Z)
• dOPSD: On-Policy Self-Distillation for Diffusion Language Models [52.60302464420127]
  Diffusion large language models generate text by iteratively denoising a masked sequence.<n>Supervised fine-tuning is off-policy and suffers from exposure bias, while reinforcement learning gives only sparse, sequence-level rewards.<n>On-policy self-distillation (OPSD) offers a promising alternative, using one model as both student and teacher.
  arXiv  Detail & Related papers  (2026-07-05T17:47:29Z)
• DOPD: Dual On-policy Distillation [59.65986569617147]
  On-policy distillation offers superior capacity transfer by supervising student-sampled trajectories with dense token-level signals.<n>We propose DOPD, an advantage-aware dual distillation paradigm that dynamically routes token-level supervision between privileged teacher and privileged student policies.<n>Experiments on both large language model (LLM) and vision-language model (VLM) settings demonstrate that DOPD consistently outperforms Vanilla OPD.
  arXiv  Detail & Related papers  (2026-06-29T17:55:53Z)
• OPID: On-Policy Skill Distillation for Agentic Reinforcement Learning [38.54413500261524]
  On-policy self-distillation offers dense token-level supervision.<n>textbfOPID (textbfOn-textbfPolicy Sktextbfill textbfDistillation) is a framework that extracts skill supervision directly from completed on trajectories.
  arXiv  Detail & Related papers  (2026-06-25T09:24:09Z)
• OmniOPSD: Rationale-Privileged On-Policy Self-Distillation for Affective Computing [62.62736813348113]
  Reinforcement learning for large language models is often hindered by severe reward sparsity in complex reasoning tasks.<n>We propose OmniOPSD, a framework that uses frontier-generated rationales as teacher-side privileged evidence.<n>Experiments on MER-UniBench show that OmniOPSD achieves state-of-the-art performance with an average score of $84.19$.
  arXiv  Detail & Related papers  (2026-06-14T16:51:22Z)
• EDGE-OPD: Internalizing Privileged Context with Evidence Guided On-Policy Distillation [5.310892696470208]
  On-Policy Distillation (OPD) has gained wide attraction as an LLM post-training paradigm.<n>The challenge in this approach is that the privileged information can change model behavior more than intended.<n>We propose EviDence GuidEd On-Policy Distillation (EDGE-OPD), a modification of OPD with two distinct characteristics.
  arXiv  Detail & Related papers  (2026-05-22T10:55:15Z)
• Harnessing LLM Agents with Skill Programs [58.356514745548026]
  We introduce HASP, a new framework that upgrades skills into executable Program Functions (PFs)<n>PFs act as executable guardrails that activate on failure-prone states and modify the next action or inject corrective context.<n> HASP drives substantial gains compared to both training-free and training-based methods on web-search, math reasoning, and coding tasks.
  arXiv  Detail & Related papers  (2026-05-18T01:35:11Z)
• PRTS: A Primitive Reasoning and Tasking System via Contrastive Representations [66.94988600664574]
  We present textbfPRTS (textbfPrimitive textbfReasoning and textbfTasking textbfSystem), a VLA foundation model that reformulates pretraining through Goal-Conditioned Reinforcement Learning.
  arXiv  Detail & Related papers  (2026-04-30T06:14:02Z)
• SELF: Self-Evolution with Language Feedback [68.6673019284853]
  'SELF' (Self-Evolution with Language Feedback) is a novel approach to advance large language models. It enables LLMs to self-improve through self-reflection, akin to human learning processes. Our experiments in mathematics and general tasks demonstrate that SELF can enhance the capabilities of LLMs without human intervention.
  arXiv  Detail & Related papers  (2023-10-01T00:52:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.