Outcome-Confounded Local Supervision in On-Policy Distillation

• URL: http://arxiv.org/abs/2607.23731v1
• Date: Sun, 26 Jul 2026 16:03:33 GMT
• Title: Outcome-Confounded Local Supervision in On-Policy Distillation
• Abstract summary: On-policy distillation trains a student on its own trajectories while a teacher supplies dense token-level likelihoods at student-visited prefixes.<n>We show that both readings are confounded by the outcome of the completed trajectory.<n>We introduce an outcome-resolved diagnostic that crosses pointwise teacher-student divergence with final-answer correctness.
• Score: 0.5937989874256571
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: On-policy distillation (OPD) trains a student on its own trajectories while a teacher supplies dense token-level likelihoods at student-visited prefixes. These likelihoods are often read locally: agreement appears safe to imitate, whereas disagreement appears to identify an error. We show that both readings are confounded by the outcome of the completed trajectory. We introduce an outcome-resolved diagnostic that crosses pointwise teacher-student divergence with final-answer correctness, separating safe imitation, productive divergence, harmful divergence, and agreement-on-failure. In an eight-seed mathematical-reasoning study with a Qwen3-8B student and Qwen3-32B teacher, agreement-on-failure constitutes 67.84% of pooled response-token mass; with a Qwen2.5-7B/32B pair it remains 67.68%. The result persists across threshold, sequence-level, format, and truncation audits. Even on prompts that the Qwen3 teacher solves in all four independent attempts, student accuracy rises to 86.91% but agreement-on-failure remains 14.76%. We then run three matched training probes that use the available signals to imitate, mask, or contrast whole trajectories; none consistently reduces agreement-on-failure. The result points to a localization limitation: local divergence paired with a trajectory-level outcome does not identify where a failed trajectory became unrecoverable. Addressing this limitation requires additional positional information, such as process labels, teacher continuations from student prefixes, or token-level alignment across rollouts. Our contribution is therefore diagnostic rather than a new training method.

Related papers

• A Token-Level Analysis of Sampled-Token Reverse-KL On-Policy Distillation [59.43605629311855]
  We analyze the gradient of the per-token K2 estimator of reverse KL with respect to the student logits.<n>Our primary contribution is therefore a gradient-level characterization of reverse-KL OPD trained with the K2 estimator, with SuRe as one empirical instantiation.
  arXiv  Detail & Related papers  (2026-08-26T11:14:42Z)
• Mismatch Matters: On-Policy Distillation Beyond Token Agreement [27.396104103072844]
  We show that mismatch tokens can be mainly categorized into two types: student-excess tokens and student-deficit tokens.<n>We propose TIDE, which applies bounded Hellinger shaping to suppress the most severe sampled excesses and an analytic teacher top-$K$ injection to restore deficient probability mass.
  arXiv  Detail & Related papers  (2026-08-10T16:53:14Z)
• WDL-OPD: Weak-Driven On-Policy Distillation via Mixture-Constrained Co-Training [48.290951447286744]
  WDL-OPD is a mixture-constrained co-training method with two trainable policies.<n>An anchor policy generates every rollout, an auxiliary policy evaluates the same visited states, and a geometric mixture of their token distributions is matched to a frozen teacher by reverse KL.<n>We show that freezing the auxiliary recovers an anchor-plus-contrast proxy target closely related to OPD$2$ and W2S-OPD, whereas joint training creates branch-level degrees of freedom that a static delta cannot express.
  arXiv  Detail & Related papers  (2026-08-10T11:22:53Z)
• BOUND: Brief-Guided Corrective Preference Distillation at Search-Control Boundaries [82.41764922522565]
  BOUND is a brief-guided corrective preference distillation framework for persistent search drift.<n>It outperforms Trajectory SFT by 5.6 EM points on Bamboogle and 4.8 accuracy points on BrowseComp-Plus.
  arXiv  Detail & Related papers  (2026-08-09T15:32:16Z)
• Trajectory-Relative Hindsight Distillation for Agentic Reinforcement Learning [22.776663901175663]
  We introduce TRIAL, a trajectory-relative hindsight distillation framework with a unified turn-aligned scoring protocol.<n>For each decision turn, TRIAL extracts an outcome view of that decision's realized consequence and evaluates the same response under ordinary and hindsight-conditioned contexts.<n>Experiments on WebShop and ALFWorld with different backbones show that TRIAL outperforms GRPO across all eight combinations of backbone, environment, and evaluation metric.
  arXiv  Detail & Related papers  (2026-08-07T16:12:58Z)
• Correcting What You Cannot See: Credit Assignment for Perception Distillation in Multimodal Reasoners [34.39191083253229]
  We introduce textbfPerception-Correction Distillation (PCD), a label-free method that identifies correctable perception failures.<n>We motivate this rule through Bayesian evidence combination and show that multiplication is the unique normalized bilinear gate that vanishes when either witness is absent.
  arXiv  Detail & Related papers  (2026-07-30T15:03:55Z)
• Pass the Baton: Trajectory-Relayed On-Policy Distillation [23.50689651536186]
  On-policy distillation grounds token-level supervision in the student's own trajectory, yet suffers from prefix failure.<n>We identify a teacher-student continuation asymmetry on failed prefixes, and convert it into a label-free handoff trigger in Relay On-Policy Distillation (Relay-OPD)<n>During training, Relay-OPD constructs relay trajectories by letting the teacher briefly take over at detected trigger points to produce a teacher leg, after which the student resumes and is optimized on the resulting trajectory.
  arXiv  Detail & Related papers  (2026-07-28T17:59:46Z)
• H$^2$SD: Hybrid Hindsight Self-Distillation [61.71131241473028]
  Reinforcement learning with verifiable rewards (RLVR) provides reliable outcome supervision for language model reasoning.<n>Existing self-distillation methods add a privileged teacher but typically assign it a fixed role.<n>We introduce Hybrid Hindsight Self-Distillation ($mathrmH2mathrmSD$), which jointly adapts teacher context and update strategy to trajectory correctness.
  arXiv  Detail & Related papers  (2026-07-21T10:47:27Z)
• Geometric Self-Distillation for Reasoning Generalization [12.38444431260744]
  On-policy distillation is a practical post-training recipe for large language models.<n>In privileged-context self-distillation, teacher and student are the same model conditioned on the same prefix.<n>We introduce GeoSD, a geometric self-distillation objective that treats this drift as movement in the student's predictive behavior.
  arXiv  Detail & Related papers  (2026-07-07T23:16:19Z)
• Benchmarking Federated Learning and Knowledge Distillation for Point Cloud Classification [5.584060970507507]
  We present a benchmark jointly evaluating federated learning (FL) and knowledge distillation (KD) for 3D point cloud classification.<n>It spans 13 FL algorithms and 10 KD objectives (a 130-pair cross-product) across 504 training runs, evaluated on ModelNet40 and a clinical craniosynostosis dataset.
  arXiv  Detail & Related papers  (2026-06-30T20:12:27Z)
• Localizing Credit at the Divergence: Path-Conditioned Self-Distillation for LLM Reasoning [12.363923974156636]
  Reinforcement learning from verifiable rewards assigns a single scalar to each rollout.<n>On-policy self-distillation addresses this by letting the same model act as a teacher conditioned on privileged information.<n>We propose Hindsight Self-Distillation (HSD), which conditions the teacher on a successful peer rollout.
  arXiv  Detail & Related papers  (2026-06-14T03:37:27Z)
• Strained Coherence: A Pre-Failure Signal in Coding Agent Execution Trajectories [0.0]
  LLM-based coding agents sometimes acknowledge a problem in their own reasoning and then proceed anyway.<n>We build a Claude Sonnet 4.6 judge that reads full trajectories and flags spans where the pattern occurs.<n>We evaluate it on 44 Terminal-bench-2 trajectories using a Qwen3.5-35B-A3B backbone.
  arXiv  Detail & Related papers  (2026-06-05T22:52:16Z)
• Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning [55.264863369127774]
  Current methods reduce each correct rollout to a single reward bit, ignoring the geometric structure shared among their hidden states.<n>We propose Hidden-Align, an auxiliary loss function that aligns the last-layer hidden states of correct rollouts at the anchor token during RL training, with zero overhead in both training and inference.<n>On eight mathematical reasoning benchmarks, Hidden-Align improves average pass@1 over the DAPO baseline by 3.8, 6.2, and 5.4 percentage points on Qwen3-1.7B, 4B, and 14B respectively, with consistent pass@k gains across all three scales.
  arXiv  Detail & Related papers  (2026-06-02T06:51:15Z)
• Bridging Reasoning Trajectories in On-Policy Distillation via Near-Future Guidance [10.04930078540686]
  On-Policy Distillation improves large language model reasoning by training a student model on trajectories sampled from its own policy under teacher supervision.<n>We show that this "trajectory-sampled but token-learned" mechanism cannot reliably bridge student trajectories toward teacher trajectories.<n>We propose Trajectory-aware OPD, which uses near-future trajectory information to identify real divergent states and distribute guidance across multiple future tokens.
  arXiv  Detail & Related papers  (2026-05-29T19:32:07Z)
• Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation [81.10000755917712]
  On-policy distillation transfers reasoning capabilities by training a student model on its own generated trajectories using token-level feedback from a teacher.<n>As student-generated prefixes lengthen, the teacher's next-token distribution becomes less confident and less discriminative.<n>To mitigate SFD, we introduce textbfLookahead Group Reward (ours).
  arXiv  Detail & Related papers  (2026-05-29T04:39:20Z)
• TCOD: Exploring Temporal Curriculum in On-Policy Distillation for Multi-turn Autonomous Agents [55.27396165691312]
  We identify a key limitation of vanilla OPD in multi-turn agent settings, which we term Trajectory-Level KL Instability.<n>We propose TCOD, a framework that controls the trajectory depth exposed to the student and progressively expands it with a curriculum schedule.<n> Experimental results across four student-teacher pairs show that TCOD mitigates KL escalation and enhances KL stability throughout training, improving agent performance by up to 18 points over vanilla OPD.
  arXiv  Detail & Related papers  (2026-04-27T03:38:27Z)
• GATES: Self-Distillation under Privileged Context with Consensus Gating [89.62339954332248]
  We study self-distillation in settings where supervision is unreliable.<n>We focus on document-grounded question answering with asymmetric context.<n>We derive supervision online from tutor consensus by sampling multiple document-grounded reasoning traces.
  arXiv  Detail & Related papers  (2026-02-24T05:56:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.