OmniOPD: Logit-Free On-Policy Distillation via Speculative Verification

• URL: http://arxiv.org/abs/2606.01476v1
• Date: Sun, 31 May 2026 22:31:15 GMT
• Title: OmniOPD: Logit-Free On-Policy Distillation via Speculative Verification
• Abstract summary: On-Policy Distillation (OPD) trains a student model on its own generative trajectories under dense token-level feedback from a stronger teacher.<n>We introduce OmniOPD, a novel framework that addresses both limitations through a logit-free, chunk-level supervision signal.
• Score: 21.095025092772257
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: On-Policy Distillation (OPD) trains a student model on its own generative trajectories under dense token-level feedback from a stronger teacher, mitigating both the off-policy distribution shift of Supervised Fine-Tuning (SFT) and the sparse credit assignment of Reinforcement Learning (RL). However, standard OPD faces two coupled limitations. First, it requires direct access to the teacher's token-level logits, excluding a broad class of capable proprietary models from serving as teachers. Second, the token-level logit signal itself is brittle, depending on a narrow overlap of plausible next tokens between teacher and student, and prone to amplifying degenerate patterns such as repetition loops. In this paper, we introduce OmniOPD, a novel framework that addresses both limitations through a logit-free, chunk-level supervision signal. OmniOPD replaces deterministic logit matching with Monte Carlo rollouts that approximate the teacher's local preferences through a continuous semantic similarity metric over multi-token chunks, and concentrates this supervision via a peak-entropy scheduler that audits the student only at its high-uncertainty reasoning forks. A Dirichlet-Multinomial Bayesian prior and a base-model KL anchor further bound the variance of discrete sampling and prevent policy collapse across unaudited tokens. Across competitive benchmarks, OmniOPD surpasses the standard OPD approach by up to +28.64% on math, confirming that chunk-level semantic verification extracts a more reliable learning signal than token-level logit matching, whose high information density is offset by significant noise and brittleness. Furthermore, when paired with stronger black-box teachers such as Claude-4.5-Haiku and Gemini-2.5-Flash, OmniOPD achieves an additional +9.54% relative on math over its open-weight teacher counterpart, advancing the student past the performance of self-exploratory RL.

Related papers

• A Token-Level Analysis of Sampled-Token Reverse-KL On-Policy Distillation [59.43605629311855]
  We analyze the gradient of the per-token K2 estimator of reverse KL with respect to the student logits.<n>Our primary contribution is therefore a gradient-level characterization of reverse-KL OPD trained with the K2 estimator, with SuRe as one empirical instantiation.
  arXiv  Detail & Related papers  (2026-08-26T11:14:42Z)
• SimpleOPD: Simple Tokenizer-Agnostic On-Policy Distillation for Long-Context Reasoning [54.402550664505576]
  On-policy distillation (OPD) offers a promising way to transfer reasoning capabilities from stronger teacher models to short-context students.<n>We study this setting by transferring proof-reasoning capabilities from the long-context reasoning model SU-01 to short-context student models.<n> Experiments on both same-family and different-family student models, including Qwen3, Qwen3.5, Intern-S2, GLM-4.7, Gemma-4, show consistent gains in mathematical reasoning.
  arXiv  Detail & Related papers  (2026-08-14T12:57:02Z)
• Teach the Magnitude, Not the Direction: Verifier-Bounded Credit Assignment for Multi-Turn Multi-step LLM Agents [15.12019413574721]
  Reinforcement learning with verifiable rewards conflates heterogeneous per-turn outcomes into a single reward signal.<n>On-policy distillation provides dense per-token supervision but is either teacher-bounded or prone to gradient concentration collapse.<n>We introduce $textbfCrEST$, a hierarchical credit assignment framework that retains RL's verifier-bounded ceiling while incorporating dense token-level signals from a privileged self-teacher.
  arXiv  Detail & Related papers  (2026-08-13T12:44:37Z)
• Distill Where You Fail: Recovering Learning Signals of Negative RL-Groups from Adaptive Teacher Guidance [50.77757355236912]
  We propose RSTG (Recovering Learning Signals via Adaptive Teacher Guidance), which applies distillation selectively and precisely where it matters most.<n> Experiments demonstrate that RSTG substantially outperforms naive GRPO+OPD by +4.02% on math and+3.05% on code.
  arXiv  Detail & Related papers  (2026-08-01T17:29:14Z)
• Lightning OPD 2.0: Mitigating Style Bias in Cross-Teacher On-Policy Distillation for Large Reasoning Models [23.269235615604245]
  We introduce Lightning OPD 2.0 with cross-fitted style residualization.<n>Across mathematical reasoning and code generation benchmarks, Lightning OPD 2.0 consistently outperforms Lightning OPD in cross-teacher settings.
  arXiv  Detail & Related papers  (2026-07-30T16:17:15Z)
• dOPSD: On-Policy Self-Distillation for Diffusion Language Models [52.60302464420127]
  Diffusion large language models generate text by iteratively denoising a masked sequence.<n>Supervised fine-tuning is off-policy and suffers from exposure bias, while reinforcement learning gives only sparse, sequence-level rewards.<n>On-policy self-distillation (OPSD) offers a promising alternative, using one model as both student and teacher.
  arXiv  Detail & Related papers  (2026-07-05T17:47:29Z)
• DOPD: Dual On-policy Distillation [59.65986569617147]
  On-policy distillation offers superior capacity transfer by supervising student-sampled trajectories with dense token-level signals.<n>We propose DOPD, an advantage-aware dual distillation paradigm that dynamically routes token-level supervision between privileged teacher and privileged student policies.<n>Experiments on both large language model (LLM) and vision-language model (VLM) settings demonstrate that DOPD consistently outperforms Vanilla OPD.
  arXiv  Detail & Related papers  (2026-06-29T17:55:53Z)
• Not All Disagreement Is Learnable: Token Teachability in On-Policy Distillation [28.572361799234784]
  We show that raw KL disagreement is a coarse proxy for learning value.<n>We formalize this local compatibility as token teachability.<n>We propose Teachability-Aware OPD, a lightweight token-position selection method.
  arXiv  Detail & Related papers  (2026-05-26T10:56:46Z)
• X-Token: Projection-Guided Cross-Tokenizer Knowledge Distillation [19.064775527374618]
  Cross-tokenizer knowledge distillation allows a student model to learn from teachers with vocabularies.<n>Logit-based methods either use only the correct-token probability, missing the full 'dark knowledge' in the teacher's distribution.<n>We propose X-Token, an approach with two complementary loss formulations targeting these issues.
  arXiv  Detail & Related papers  (2026-05-20T19:59:31Z)
• Self-Distilled Agentic Reinforcement Learning [65.24201057390938]
  Self-Distilled Agentic Reinforcement Learning maps detached token-level signals into a sigmoid gate.<n>SDAR consistently outperforms hybrid RL--OPSD baselines across model scales.
  arXiv  Detail & Related papers  (2026-05-14T17:51:26Z)
• Prefix Teach, Suffix Fade: Local Teachability Collapse in Strong-to-Weak On-Policy Distillation [49.117085054884676]
  On-policy distillation trains a student model on its own rollouts using dense feedback from a stronger teacher.<n>We operationalize this principle through a trajectory-specific release rule.<n> Experimental results across strong-to-weak distillation tasks indicate that this release rule consistently outperforms standard full-trajectory OPD.
  arXiv  Detail & Related papers  (2026-05-13T15:05:30Z)
• The Extrapolation Cliff in On-Policy Distillation of Near-Deterministic Structured Outputs [52.709361620508595]
  ListOPD brings a student to in-domain with an 8B-SFT baseline at one-fifth the parameters.<n>On Amazon Fashion, three pre-registered tests--a fine-grid cliff interval test, and a small-clip cross-prediction--fall within their locked prediction windows--with the small-clip value matching the closed-form prediction below grid resolution.
  arXiv  Detail & Related papers  (2026-05-09T06:48:00Z)
• SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillation [39.59759394378816]
  On-policy distillation assumes teacher and student predictions are comparable token by token.<n>We propose SimCT, which restores this signal by enlarging the supervision space.<n>SimCT shows consistent gains over shared-vocabulary OPD and representative cross-tokenizer baselines.
  arXiv  Detail & Related papers  (2026-05-08T13:16:17Z)
• PAINT: Partial-Solution Adaptive Interpolated Training for Self-Distilled Reasoners [9.324642081509756]
  Large language model (LLM) reasoning requires supervision that is both aligned with the model's own test-time states and informative at the token level.<n>Recent privileged on-policy self-distillation explores a middle ground by scoring student rollouts with the same model under verified solution context.<n>We propose PAINT, which masks the verified solution according to rollout-reference overlap and applies a small energy-space on a sparse set of entropy-mismatch token positions.
  arXiv  Detail & Related papers  (2026-04-29T11:56:07Z)
• Self-Distilled RLVR [57.37526213765131]
  We show that learning signals solely derived from the privileged teacher result in severe information leakage and unstable long-term training.<n>We propose textbfRLSD (textbfRLVR with textbfSelf-textbfDistillation)<n>This enables RLSD to simultaneously harness the strengths of both RLVR and OPSD, achieving a higher convergence ceiling and superior training stability.
  arXiv  Detail & Related papers  (2026-04-03T15:50:07Z)
• Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes [31.95045602299568]
  On-policy distillation (OPD) is appealing for large language model (LLM) post-training because it evaluates teacher feedback on student-generated rollouts rather than fixed teacher traces.<n>We revisit OPD from the estimator and implementation sides.<n>We identify three failure modes of sampled-token OPD: an imbalanced one-token signal, unreliable teacher guidance on student-generated prefixes, and distortions caused by tokenizer or special-token mismatch.
  arXiv  Detail & Related papers  (2026-03-26T15:35:59Z)
• Prototypical Classifier for Robust Class-Imbalanced Learning [64.96088324684683]
  We propose textitPrototypical, which does not require fitting additional parameters given the embedding network. Prototypical produces balanced and comparable predictions for all classes even though the training set is class-imbalanced. We test our method on CIFAR-10LT, CIFAR-100LT and Webvision datasets, observing that Prototypical obtains substaintial improvements compared with state of the arts.
  arXiv  Detail & Related papers  (2021-10-22T01:55:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.