Is More Privileged Information Better? From Solution Traces to Problem-Solving Structure in Self-Distilled Reasoning

• URL: http://arxiv.org/abs/2608.01589v1
• Date: Mon, 03 Aug 2026 01:47:25 GMT
• Title: Is More Privileged Information Better? From Solution Traces to Problem-Solving Structure in Self-Distilled Reasoning
• Abstract summary: On-policy self-distillation improves reasoning by using a privileged view of a model conditioned on reference solutions to supervise a student view that observes only the question.<n>We propose Problem-Space-Guided OPSD (PS-OPSD), which replaces the complete solution with trajectory-grounded guidance describing the initial state, goal conditions, constraints, and a selected state-transition path.
• Score: 17.645367206348222
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: On-policy self-distillation (OPSD) improves reasoning by using a privileged view of a model conditioned on reference solutions to supervise a student view that observes only the question. However, the teacher-provided token-level targets may depend on reference-specific information unavailable at inference time. We propose Problem-Space-Guided OPSD (PS-OPSD), which replaces the complete solution with trajectory-grounded guidance describing the initial state, goal conditions, constraints, and a selected state-transition path. The student rollout and OPSD objective remain unchanged. Across three mathematical reasoning benchmarks and model scales ranging from 1.7B to 8B, PS-OPSD achieves the highest aggregate question-only accuracy among the compared methods. Controlled experiments further indicate that guidance relevance and path coherence contribute to these gains, highlighting the representation of privileged information as an important design choice in OPSD.

Related papers

• Rethinking Privileged Information in On-Policy Self-Distillation [45.88028371034407]
  On-policy self-distillation (OPSD) trains a student on its own responses using token-level supervision from the same model conditioned on privileged reference information.<n>We investigate whether performance gains from OPSD show that the student learned the information in the reference or instead reflect recovery of reasoning behavior already present in the base model.
  arXiv  Detail & Related papers  (2026-08-18T19:42:16Z)
• Latent On-Policy Self-Distillation [55.57800223145407]
  On-policy self-distillation (OPSD) offers an effective pathway by using a privileged self-teacher to provide dense supervision on the student's own trajectories.<n>Existing methods still rely heavily on designer-specified privileged artifacts.<n>We introduce Latent On-Policy Self-Distillation (LOPD), which makes the teacher's privileged context itself learnable end-to-end from experience.
  arXiv  Detail & Related papers  (2026-08-13T10:05:51Z)
• Advantage-Guided Gate: Reshaping Open-Ended Reasoning for Vision-Based Spatial Intelligence [70.8951332710039]
  Multimodal large language models (MLLMs) have demonstrated significant potential in complex spatial scene understanding and reasoning tasks.<n>However, their open-ended reasoning process is prone to decision errors and error accumulation, leading to instability in answer quality.<n>We propose an advantage-guided gating framework that dynamically intervenes in and corrects deviations during the reasoning process.
  arXiv  Detail & Related papers  (2026-08-08T07:46:04Z)
• dOPSD: On-Policy Self-Distillation for Diffusion Language Models [52.60302464420127]
  Diffusion large language models generate text by iteratively denoising a masked sequence.<n>Supervised fine-tuning is off-policy and suffers from exposure bias, while reinforcement learning gives only sparse, sequence-level rewards.<n>On-policy self-distillation (OPSD) offers a promising alternative, using one model as both student and teacher.
  arXiv  Detail & Related papers  (2026-07-05T17:47:29Z)
• DemoPSD: Disagreement-Modulated Policy Self-Distillation [49.30809363804017]
  On-policy self-distillation has emerged as a practical method for training large language models.<n>DemoPSD steers the student toward a weighted geometric combination of the teacher and student distributions.
  arXiv  Detail & Related papers  (2026-07-02T17:58:29Z)
• Purified OPSD: On-Policy Self-Distillation Without Losing How to Think [52.879387744920415]
  We find that on-policy self-distillation (OPSD) consistently fails on long chain-of-thought (long-CoT) reasoning models.<n>We propose a two-step solution: first, we construct a reference-only teacher to isolate the non-transferable component of the supervision signal.<n>Second, we use pointwise mutual information (PMI) as the mechanism to transform this residual into a well-formed PMI target distribution.
  arXiv  Detail & Related papers  (2026-07-02T14:33:07Z)
• Rethinking Reward Supervision: Rubric-Conditioned Self-Distillation [60.55792673956761]
  We propose textbfRubric-Conditioned Self-Distillation, a framework that incorporates rubrics as structured, fine-grained feedback for on-policy self-distillation.<n>We evaluate on a diverse suite of science reasoning benchmarks and results show that rubric-conditioned self-distillation effectively converts rubric-level criteria into token-level guidance.
  arXiv  Detail & Related papers  (2026-06-17T17:54:04Z)
• Teaching the Way, Not the Answer: Privileged Tutoring Distillation for Multimodal Policy Optimization [22.479229036646156]
  We propose a Privileged Tutoring Distillation Policy Optimization framework for Reinforcement Learning with Verifiable Rewards (RLVR)<n> PTD-PO provides dense guidance without exposing the answer to the student policy.<n> Experiments on LVLMs ranging from 2B to 8B parameters show that PTD-PO consistently outperforms RLVR and distillation baselines.
  arXiv  Detail & Related papers  (2026-06-05T07:43:22Z)
• EDGE-OPD: Internalizing Privileged Context with Evidence Guided On-Policy Distillation [5.310892696470208]
  On-Policy Distillation (OPD) has gained wide attraction as an LLM post-training paradigm.<n>The challenge in this approach is that the privileged information can change model behavior more than intended.<n>We propose EviDence GuidEd On-Policy Distillation (EDGE-OPD), a modification of OPD with two distinct characteristics.
  arXiv  Detail & Related papers  (2026-05-22T10:55:15Z)
• OGLS-SD: On-Policy Self-Distillation with Outcome-Guided Logit Steering for LLM Reasoning [19.98950359294245]
  We study on-policy self-distillation (OPSD), where a language model improves its reasoning ability by distilling privileged teacher distributions along its own on-policy trajectories.<n>Despite the performance gains of OPSD, we identify a common but often overlooked mismatch between teacher and student responses.
  arXiv  Detail & Related papers  (2026-05-12T17:00:53Z)
• Optimizing In-Context Demonstrations for LLM-based Automated Grading [31.353360036776976]
  GUIDE (Grading Using Iteratively Designed Exemplars) is a framework that reframes exemplar selection and refinement as a boundary-focused optimization problem.<n>We show that GUIDE significantly outperforms standard retrieval baselines in experiments in physics, chemistry, and pedagogical content knowledge.
  arXiv  Detail & Related papers  (2026-02-28T04:52:38Z)
• Mitigating Spurious Correlations Between Question and Answer via Chain-of-Thought Correctness Perception Distillation [25.195244084313114]
  Chain-of-Thought Correctness Perception Distillation (CoPeD) aims to improve the reasoning quality of the student model.<n>CoPeD encourages the student model to predict answers based on correct rationales and revise them when they are incorrect.
  arXiv  Detail & Related papers  (2025-09-06T05:33:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.