Localizing Credit at the Divergence: Path-Conditioned Self-Distillation for LLM Reasoning

• URL: http://arxiv.org/abs/2606.15576v1
• Date: Sun, 14 Jun 2026 03:37:27 GMT
• Title: Localizing Credit at the Divergence: Path-Conditioned Self-Distillation for LLM Reasoning
• Abstract summary: Reinforcement learning from verifiable rewards assigns a single scalar to each rollout.<n>On-policy self-distillation addresses this by letting the same model act as a teacher conditioned on privileged information.<n>We propose Hindsight Self-Distillation (HSD), which conditions the teacher on a successful peer rollout.
• Score: 12.363923974156636
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement learning from verifiable rewards assigns a single scalar to each rollout, leaving token-level credit assignment underspecified in long reasoning traces. On-policy self-distillation addresses this by letting the same model act as a teacher conditioned on privileged information, producing a dense per-token signal. But the common choice of a ground-truth answer is only an endpoint cue: on terse-answer tasks, the teacher falls silent at the intermediate positions where path-level guidance matters most. We propose Hindsight Self-Distillation (HSD), which conditions the teacher on a successful peer rollout drawn from the current training group. Such a peer is an exact sample from the success-conditioned policy, requiring no additional sampled rollouts. By providing a full successful continuation rather than only the final answer, the resulting credit signal concentrates at the divergence position between a failed rollout and a successful peer. Across Qwen3-8B and Qwen3-32B on math and code benchmarks, HSD obtains the best result against GRPO variants and on-policy distillation baselines, with the largest gains on terse-answer tasks such as AIME.

Related papers

• Privileged, but Biased: How PI-Conditioned Teachers Break Self-Distillation [3.5592415157621105]
  Self-distillation (SD) has emerged as a compute-efficient alternative to reinforcement learning with verifiable rewards.<n>We reproduce SDPO's reported gains in its easy setting, then apply the identical setup to difficult tasks and find that it does not.<n>We explain this failure through a single causal chain from the loss to the model it produces.
  arXiv  Detail & Related papers  (2026-08-05T12:59:10Z)
• Distill Where You Fail: Recovering Learning Signals of Negative RL-Groups from Adaptive Teacher Guidance [50.77757355236912]
  We propose RSTG (Recovering Learning Signals via Adaptive Teacher Guidance), which applies distillation selectively and precisely where it matters most.<n> Experiments demonstrate that RSTG substantially outperforms naive GRPO+OPD by +4.02% on math and+3.05% on code.
  arXiv  Detail & Related papers  (2026-08-01T17:29:14Z)
• H$^2$SD: Hybrid Hindsight Self-Distillation [61.71131241473028]
  Reinforcement learning with verifiable rewards (RLVR) provides reliable outcome supervision for language model reasoning.<n>Existing self-distillation methods add a privileged teacher but typically assign it a fixed role.<n>We introduce Hybrid Hindsight Self-Distillation ($mathrmH2mathrmSD$), which jointly adapts teacher context and update strategy to trajectory correctness.
  arXiv  Detail & Related papers  (2026-07-21T10:47:27Z)
• Rethinking Reward Supervision: Rubric-Conditioned Self-Distillation [60.55792673956761]
  We propose textbfRubric-Conditioned Self-Distillation, a framework that incorporates rubrics as structured, fine-grained feedback for on-policy self-distillation.<n>We evaluate on a diverse suite of science reasoning benchmarks and results show that rubric-conditioned self-distillation effectively converts rubric-level criteria into token-level guidance.
  arXiv  Detail & Related papers  (2026-06-17T17:54:04Z)
• Learning from Own Solutions: Self-Conditioned Credit Assignment for Reinforcement Learning with Verifiable Rewards [49.1203423784326]
  We propose SC- GRPO (Self-Conditioned GRPO), which uses KL divergence mentioned before as a multiplicative weight on GRPO gradients.<n>Across five benchmarks spanning math, code, and agentic tasks, SC- GRPO consistently outperforms 8.1% over GRPO and 5.9% over DAPO with stronger OOD performance.
  arXiv  Detail & Related papers  (2026-06-17T08:26:02Z)
• HERO: Hindsight-Enhanced Reflection from Environment Observations for Agentic Self-Distillation [50.53459634301361]
  HERO is a hindsight-enhanced self-distillation framework that uses next environment observations as locally aligned feedback.<n> HERO improves task success and reduces unnecessary turns over environment-feedback-only self-distillation and GRPO.
  arXiv  Detail & Related papers  (2026-06-10T01:35:34Z)
• PBSD: Privileged Bayesian Self-Distillation for Long-Horizon Credit Assignment [35.978805768172656]
  Long-horizon agentic tasks pose a fundamental credit assignment challenge for outcome-base reinforcement learning.<n>We propose PBSD (Privileged Bayesian Self-Distillation), a Bayes-calibrated self-distillation method for fine-grained credit assignment under sparse final rewards.
  arXiv  Detail & Related papers  (2026-06-08T11:20:58Z)
• Tailoring Teaching to Aptitude: Direction-Adaptive Self-Distillation for LLM Reasoning [41.384652481442735]
  We propose textbfDirection-Adaptive Self-Distillation (textbfDASD), which reframes privileged self-distillation from uniform teacher imitation into entropy-routed directional supervision.<n>Across six mathematical reasoning benchmarks, DASD achieves the best macro Avg@16 over strong RLVR and self-distillation baselines.
  arXiv  Detail & Related papers  (2026-05-21T10:07:46Z)
• CEPO: RLVR Self-Distillation using Contrastive Evidence Policy Optimization [50.59956036193097]
  A model produces a correct solution under reinforcement learning with verifiable rewards (RLVR)<n>Every token receives the same reward signal regardless of whether it was a decisive reasoning step or a grammatical filler.<n>We propose Contrastive Evidence Policy Optimization (CEPO)<n>CEPO asks a sharper question at every token: "does the correct answer favor this token?" but "does the correct answer favor it while the wrong answer disfavors it?"
  arXiv  Detail & Related papers  (2026-05-19T06:46:19Z)
• Prefix Teach, Suffix Fade: Local Teachability Collapse in Strong-to-Weak On-Policy Distillation [49.117085054884676]
  On-policy distillation trains a student model on its own rollouts using dense feedback from a stronger teacher.<n>We operationalize this principle through a trajectory-specific release rule.<n> Experimental results across strong-to-weak distillation tasks indicate that this release rule consistently outperforms standard full-trajectory OPD.
  arXiv  Detail & Related papers  (2026-05-13T15:05:30Z)
• Learning with Rare Success but Rich Feedback via Reflection-Enhanced Self-Distillation [71.16738415436458]
  We introduce Reflection-Enhanced Self-Distillation (RESD), a framework that transforms raw failure feedback into an active source of corrective supervision.<n>RESD interprets failed trajectories by generating retrospective reflections to diagnose local errors, and curates a persistent global playbook.<n> Empirical evaluations on multiple continual learning tasks demonstrate that RESD substantially outperforms standard self-distillation baselines.
  arXiv  Detail & Related papers  (2026-05-12T20:46:05Z)
• Multi-Rollout On-Policy Distillation via Peer Successes and Failures [31.731266689209914]
  Large language models are often post-trained with sparse verifier rewards, which indicate whether a sampled trajectory succeeds but provide limited guidance about where reasoning succeeds or fails.<n>On-policy distillation (OPD) offers denser token-level supervision by training on student-generated trajectories.<n>We introduce Multi-Rollout On-Policy Distillation (MOPD), a peer-conditioned distillation framework that uses the student's local rollout group to construct more informative teacher signals.
  arXiv  Detail & Related papers  (2026-05-12T18:57:44Z)
• OGLS-SD: On-Policy Self-Distillation with Outcome-Guided Logit Steering for LLM Reasoning [19.98950359294245]
  We study on-policy self-distillation (OPSD), where a language model improves its reasoning ability by distilling privileged teacher distributions along its own on-policy trajectories.<n>Despite the performance gains of OPSD, we identify a common but often overlooked mismatch between teacher and student responses.
  arXiv  Detail & Related papers  (2026-05-12T17:00:53Z)
• PAINT: Partial-Solution Adaptive Interpolated Training for Self-Distilled Reasoners [9.324642081509756]
  Large language model (LLM) reasoning requires supervision that is both aligned with the model's own test-time states and informative at the token level.<n>Recent privileged on-policy self-distillation explores a middle ground by scoring student rollouts with the same model under verified solution context.<n>We propose PAINT, which masks the verified solution according to rollout-reference overlap and applies a small energy-space on a sparse set of entropy-mismatch token positions.
  arXiv  Detail & Related papers  (2026-04-29T11:56:07Z)
• Verified Critical Step Optimization for LLM Agents [67.05296684575445]
  Critical Step Optimization focuses preference learning on verified critical steps.<n>Method starts from failed policy trajectories rather than expert demonstrations.<n>Experiments on GAIA-Text-103 and XBench-DeepSearch show that CSO achieves 37% and 26% relative improvement over the SFT baseline.
  arXiv  Detail & Related papers  (2026-02-03T11:41:02Z)
• Learning from Diverse Reasoning Paths with Routing and Collaboration [65.77515749498575]
  We propose Quality-filtered Routing with Cooperative Distillation (QR-Distill), combining path quality filtering, conditional routing, and peer teaching.<n> Experiments demonstrate QR-Distill's superiority over traditional single- and multi-path distillation methods.
  arXiv  Detail & Related papers  (2025-08-23T01:15:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.