DAPD: Dual-Anchored Policy Distillation

• URL: http://arxiv.org/abs/2608.01735v1
• Date: Mon, 03 Aug 2026 06:00:44 GMT
• Title: DAPD: Dual-Anchored Policy Distillation
• Abstract summary: On-policy (self) distillation (OPSD) is increasingly adopted for language-model post-training.<n>It strengthens the teacher with privileged information but can induce a privilege illusion.<n>We propose Dual-Anchored Policy Distillation (DAPD) to resolve this asymmetry.
• Score: 23.600341643801844
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: On-policy (self) distillation (OPSD) is increasingly adopted for language-model post-training. It strengthens the teacher with privileged information but can induce a privilege illusion: the student learns privilege-dependent behavior it cannot reproduce from its inference-time context, yet behaves as if the training-time privileged information remained available, ultimately degrading performance. In this paper, we identify information asymmetry between the privileged teacher and the student at inference as the root cause of this failure in OPSD. To resolve this asymmetry, we propose Dual-Anchored Policy Distillation (DAPD), a unified framework with two levels of anchoring. Dual-Path Anchoring (DPA) introduces a self-conditioned bridge and aligns reference and rollout behavior along two matched-information paths, preventing privilege-dependent behavior from being transferred to the inference-time student. Dual-Source Anchoring (DSA) applies these paths in both reference-to-rollout and rollout-to-reference directions, reducing reliance on privileged reference guidance while preserving correctness supervision. Extensive experiments show that DAPD significantly alleviates privilege illusion, outperforming OPSD on Qwen3-4B by +2.00 points on average across tasks. Notably, its gains persist across scales, reaching +2.69 at 4B and +2.78 at 32B.

Related papers

• Latent On-Policy Self-Distillation [55.57800223145407]
  On-policy self-distillation (OPSD) offers an effective pathway by using a privileged self-teacher to provide dense supervision on the student's own trajectories.<n>Existing methods still rely heavily on designer-specified privileged artifacts.<n>We introduce Latent On-Policy Self-Distillation (LOPD), which makes the teacher's privileged context itself learnable end-to-end from experience.
  arXiv  Detail & Related papers  (2026-08-13T10:05:51Z)
• Adaptive Supervised Anchoring for On-Policy Self-Distillation [15.445625053566244]
  On-policy self-distillation adapts a language model by distilling guidance from a frozen teacher on trajectories sampled from the student.<n>We show that when student rollouts drift from target trajectories, conditioning the teacher on off-target prefixes substantially weakens its task-relevant supervision.<n>We propose a unified training framework that separates two complementary supervision pathways.
  arXiv  Detail & Related papers  (2026-08-08T05:46:32Z)
• dOPSD: On-Policy Self-Distillation for Diffusion Language Models [52.60302464420127]
  Diffusion large language models generate text by iteratively denoising a masked sequence.<n>Supervised fine-tuning is off-policy and suffers from exposure bias, while reinforcement learning gives only sparse, sequence-level rewards.<n>On-policy self-distillation (OPSD) offers a promising alternative, using one model as both student and teacher.
  arXiv  Detail & Related papers  (2026-07-05T17:47:29Z)
• DemoPSD: Disagreement-Modulated Policy Self-Distillation [49.30809363804017]
  On-policy self-distillation has emerged as a practical method for training large language models.<n>DemoPSD steers the student toward a weighted geometric combination of the teacher and student distributions.
  arXiv  Detail & Related papers  (2026-07-02T17:58:29Z)
• Purified OPSD: On-Policy Self-Distillation Without Losing How to Think [52.879387744920415]
  We find that on-policy self-distillation (OPSD) consistently fails on long chain-of-thought (long-CoT) reasoning models.<n>We propose a two-step solution: first, we construct a reference-only teacher to isolate the non-transferable component of the supervision signal.<n>Second, we use pointwise mutual information (PMI) as the mechanism to transform this residual into a well-formed PMI target distribution.
  arXiv  Detail & Related papers  (2026-07-02T14:33:07Z)
• DOPD: Dual On-policy Distillation [59.65986569617147]
  On-policy distillation offers superior capacity transfer by supervising student-sampled trajectories with dense token-level signals.<n>We propose DOPD, an advantage-aware dual distillation paradigm that dynamically routes token-level supervision between privileged teacher and privileged student policies.<n>Experiments on both large language model (LLM) and vision-language model (VLM) settings demonstrate that DOPD consistently outperforms Vanilla OPD.
  arXiv  Detail & Related papers  (2026-06-29T17:55:53Z)
• Beyond Absolute Imitation: Anchored Residual Guidance for Privileged On-Policy Distillation [9.745547158215905]
  We introduce Anchored Residual On-Policy Distillation (AR-OPD), a dual-view framework that disentangles privileged supervision.<n>AR-OPD outperforms full privileged OPD by 2.3 points and SFT by 7.9 points across diverse reasoning tasks.
  arXiv  Detail & Related papers  (2026-06-09T03:51:41Z)
• EDGE-OPD: Internalizing Privileged Context with Evidence Guided On-Policy Distillation [5.310892696470208]
  On-Policy Distillation (OPD) has gained wide attraction as an LLM post-training paradigm.<n>The challenge in this approach is that the privileged information can change model behavior more than intended.<n>We propose EviDence GuidEd On-Policy Distillation (EDGE-OPD), a modification of OPD with two distinct characteristics.
  arXiv  Detail & Related papers  (2026-05-22T10:55:15Z)
• Self-Distilled Agentic Reinforcement Learning [65.24201057390938]
  Self-Distilled Agentic Reinforcement Learning maps detached token-level signals into a sigmoid gate.<n>SDAR consistently outperforms hybrid RL--OPSD baselines across model scales.
  arXiv  Detail & Related papers  (2026-05-14T17:51:26Z)
• The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes [10.319573084070578]
  On-policy distillation (OPD) and on-policy self-distillation (OPSD) have emerged as promising post-training methods for large language models.<n>We present a comprehensive empirical study of when OPD and OPSD work, when they fail, and why.
  arXiv  Detail & Related papers  (2026-05-11T19:44:59Z)
• SCOPE: Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adaptive Weighting [17.504616835765617]
  We propose a dual-path adaptive training framework that routes on-policy rollouts by correctness into two complementary supervision paths.<n>SCOPE achieves an average relative improvement of 11.42% in Avg@32 and 7.30% in Pass@32 over competitive baselines.
  arXiv  Detail & Related papers  (2026-04-12T15:26:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.