Related papers: OGLS-SD: On-Policy Self-Distillation with Outcome-Guided Logit Steering for LLM Reasoning

Related papers

Adaptive Supervised Anchoring for On-Policy Self-Distillation [15.445625053566244]
On-policy self-distillation adapts a language model by distilling guidance from a frozen teacher on trajectories sampled from the student.<n>We show that when student rollouts drift from target trajectories, conditioning the teacher on off-target prefixes substantially weakens its task-relevant supervision.<n>We propose a unified training framework that separates two complementary supervision pathways.
arXiv Detail & Related papers (2026-08-08T05:46:32Z)
Group-Reflective Self-Distillation for Agentic Reinforcement Learning [8.041052886037766]
Group-Reflective Self-Distillation (GRSD) derives capability-aligned and outcome-discriminative guidance from the policy's own verified rollouts.<n>GRSD consistently outperforms competitive baselines and generalizes more effectively to unseen tasks.
arXiv Detail & Related papers (2026-07-30T11:49:46Z)
H$^2$SD: Hybrid Hindsight Self-Distillation [61.71131241473028]
Reinforcement learning with verifiable rewards (RLVR) provides reliable outcome supervision for language model reasoning.<n>Existing self-distillation methods add a privileged teacher but typically assign it a fixed role.<n>We introduce Hybrid Hindsight Self-Distillation ($mathrmH2mathrmSD$), which jointly adapts teacher context and update strategy to trajectory correctness.
arXiv Detail & Related papers (2026-07-21T10:47:27Z)
dOPSD: On-Policy Self-Distillation for Diffusion Language Models [52.60302464420127]
Diffusion large language models generate text by iteratively denoising a masked sequence.<n>Supervised fine-tuning is off-policy and suffers from exposure bias, while reinforcement learning gives only sparse, sequence-level rewards.<n>On-policy self-distillation (OPSD) offers a promising alternative, using one model as both student and teacher.
arXiv Detail & Related papers (2026-07-05T17:47:29Z)
Purified OPSD: On-Policy Self-Distillation Without Losing How to Think [52.879387744920415]
We find that on-policy self-distillation (OPSD) consistently fails on long chain-of-thought (long-CoT) reasoning models.<n>We propose a two-step solution: first, we construct a reference-only teacher to isolate the non-transferable component of the supervision signal.<n>Second, we use pointwise mutual information (PMI) as the mechanism to transform this residual into a well-formed PMI target distribution.
arXiv Detail & Related papers (2026-07-02T14:33:07Z)
Rethinking Reward Supervision: Rubric-Conditioned Self-Distillation [60.55792673956761]
We propose textbfRubric-Conditioned Self-Distillation, a framework that incorporates rubrics as structured, fine-grained feedback for on-policy self-distillation.<n>We evaluate on a diverse suite of science reasoning benchmarks and results show that rubric-conditioned self-distillation effectively converts rubric-level criteria into token-level guidance.
arXiv Detail & Related papers (2026-06-17T17:54:04Z)
Localizing Credit at the Divergence: Path-Conditioned Self-Distillation for LLM Reasoning [12.363923974156636]
Reinforcement learning from verifiable rewards assigns a single scalar to each rollout.<n>On-policy self-distillation addresses this by letting the same model act as a teacher conditioned on privileged information.<n>We propose Hindsight Self-Distillation (HSD), which conditions the teacher on a successful peer rollout.
arXiv Detail & Related papers (2026-06-14T03:37:27Z)
Tailoring Teaching to Aptitude: Direction-Adaptive Self-Distillation for LLM Reasoning [41.384652481442735]
We propose textbfDirection-Adaptive Self-Distillation (textbfDASD), which reframes privileged self-distillation from uniform teacher imitation into entropy-routed directional supervision.<n>Across six mathematical reasoning benchmarks, DASD achieves the best macro Avg@16 over strong RLVR and self-distillation baselines.
arXiv Detail & Related papers (2026-05-21T10:07:46Z)
Prefix Teach, Suffix Fade: Local Teachability Collapse in Strong-to-Weak On-Policy Distillation [49.117085054884676]
On-policy distillation trains a student model on its own rollouts using dense feedback from a stronger teacher.<n>We operationalize this principle through a trajectory-specific release rule.<n> Experimental results across strong-to-weak distillation tasks indicate that this release rule consistently outperforms standard full-trajectory OPD.
arXiv Detail & Related papers (2026-05-13T15:05:30Z)
Teacher-Guided Policy Optimization for LLM Distillation [90.49982387646861]
Teacher-Guided Policy Optimization (TGPO) is an on-policy algorithm that incorporates dense directional guidance by leveraging teacher predictions conditioned on the student's rollout.<n>Experiments on complex reasoning benchmarks demonstrate that TGPO significantly outperforms standard baselines and is robust to different teachers.
arXiv Detail & Related papers (2026-05-13T09:20:03Z)
On-Policy Distillation with Best-of-N Teacher Rollout Selection [54.91780727674628]
We propose BRTS, a Best-of-N Rollout Teacher Selection framework for on-policy distillation.<n>BRTS augments standard student-context OPD with a teacher-context supervision branch constructed from the teacher trajectory.<n>BRTS improves over standard OPD on challenging reasoning benchmarks, with the largest gains on harder datasets.
arXiv Detail & Related papers (2026-05-10T19:49:00Z)
Preference-Based Self-Distillation: Beyond KL Matching via Reward Regularization [18.027254451537342]
Existing self-distillation methods largely reduce learning to KL matching toward the context-augmented teacher model.<n>We propose textbfPreference-textbfBased textbfSelf-textbfDistillation (textbfPBSD), which revisits on-policy self-distillation through a reward-regularized perspective.
arXiv Detail & Related papers (2026-05-06T15:31:50Z)
The Illusion of Certainty: Decoupling Capability and Calibration in On-Policy Distillation [67.26315138466312]
We propose a calibration-aware OPD framework, CaOPD, that estimates empirical confidence from model rollouts.<n>Our findings highlight that capability distillation does not imply confidence, and that confidence should be treated as an essential objective in post-training.
arXiv Detail & Related papers (2026-04-18T04:43:40Z)
SCOPE: Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adaptive Weighting [17.504616835765617]
We propose a dual-path adaptive training framework that routes on-policy rollouts by correctness into two complementary supervision paths.<n>SCOPE achieves an average relative improvement of 11.42% in Avg@32 and 7.30% in Pass@32 over competitive baselines.
arXiv Detail & Related papers (2026-04-12T15:26:14Z)
Beyond Hard Negatives: The Importance of Score Distribution in Knowledge Distillation for Dense Retrieval [26.186599286240362]
We propose a Stratified Sampling strategy that uniformly covers the entire score spectrum.<n>Experiments confirm that Stratified Sampling, which preserves the variance and entropy of teacher scores, serves as a robust baseline.
arXiv Detail & Related papers (2026-04-06T15:02:35Z)
KEPO: Knowledge-Enhanced Preference Optimization for Reinforcement Learning with Reasoning [24.072603982041798]
Reinforcement learning has emerged as a promising paradigm for inducing explicit reasoning behaviors in large language and vision-language models.<n>However, reasoning-oriented RL post-training remains fundamentally challenging due to sparse trajectory-level rewards.<n>Recent on-policy distillation methods introduce dense teacher supervision to stabilize optimization, but apply it uniformly across all generated trajectories.
arXiv Detail & Related papers (2026-01-30T23:28:37Z)
Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models [44.041109669153506]
On-Policy Self-Distillation (OPSD) is a framework where a single model acts as both teacher and student by conditioning on different contexts.<n>We demonstrate the efficacy of our method on multiple mathematical reasoning benchmarks.
arXiv Detail & Related papers (2026-01-26T17:56:50Z)
AdaSwitch: Adaptive Switching Generation for Knowledge Distillation [58.647880811071495]
Small language models (SLMs) are crucial for applications with strict latency and computational constraints.<n>We propose AdaSwitch, a novel approach that combines on-policy and off-policy generation at the token level.<n>AdaSwitch consistently improves accuracy, offering a practical and effective method for distilling SLMs with acceptable additional overhead.
arXiv Detail & Related papers (2025-10-09T06:38:37Z)
Can Large Reasoning Models Self-Train? [51.0277533541394]
We use majority voting as a simple self-feedback mechanism to study whether self-training can be sustained within reinforcement learning.<n>We find that this basic approach improves not only the model's reasoning performance, but also its capability of generating better quality feedback for the next RL iteration.<n>Yet our analysis also reveals a critical limitation of such a self-training paradigm - prolonged RL with self-reward leads to reward hacking, resulting in sudden and complete performance collapse.
arXiv Detail & Related papers (2025-05-27T17:16:00Z)
Learning Domain Adaptive Object Detection with Probabilistic Teacher [93.76128726257946]
We present a simple yet effective framework, termed as Probabilistic Teacher (PT) PT aims to capture the uncertainty of unlabeled target data from a gradually evolving teacher and guides the learning of a student in a mutually beneficial manner. We also present a novel Entropy Focal Loss (EFL) to further facilitate the uncertainty-guided self-training.
arXiv Detail & Related papers (2022-06-13T16:24:22Z)
Improving Self-supervised Learning with Automated Unsupervised Outlier Arbitration [83.29856873525674]
We introduce a lightweight latent variable model UOTA, targeting the view sampling issue for self-supervised learning. Our method directly generalizes to many mainstream self-supervised learning approaches.
arXiv Detail & Related papers (2021-12-15T14:05:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.