Geometric Self-Distillation for Reasoning Generalization

• URL: http://arxiv.org/abs/2607.06855v1
• Date: Tue, 07 Jul 2026 23:16:19 GMT
• Title: Geometric Self-Distillation for Reasoning Generalization
• Abstract summary: On-policy distillation is a practical post-training recipe for large language models.<n>In privileged-context self-distillation, teacher and student are the same model conditioned on the same prefix.<n>We introduce GeoSD, a geometric self-distillation objective that treats this drift as movement in the student's predictive behavior.
• Score: 12.38444431260744
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: On-policy distillation is a practical post-training recipe for large language models, supplying dense teacher supervision on the student's own trajectories. In privileged-context self-distillation, teacher and student are the same model conditioned on the same prefix, but the teacher also sees a hint or the full solution trace. This makes supervision abundant but harder to trust: the teacher can be confident about continuations its privileged view makes obvious but the student cannot yet justify. The distillation pull is strongest where teacher and student disagree most, and over many updates it accumulates into drift that degrades out-of-distribution (OOD) reasoning. We introduce GeoSD, a geometric self-distillation objective that treats this drift as movement in the student's predictive behavior and counters it in two complementary ways. A Hellinger loss scales each teacher preference by the overlap the student already shares with it, attenuating the pull on tokens the student cannot yet support. Since these pulls still compound over training, a proximal term penalizes how far the student's predictions drift from a recent checkpoint, measured as a Fisher-Rao distance. Both are distances in the same geometry of next-token distributions, and a natural-gradient update takes its steps in that geometry rather than in parameter space. Across mathematical reasoning benchmarks and three model families, GeoSD preserves the in-distribution gains of self-distillation while improving average OOD accuracy by 5.7-8.6 points over the base model, with gains holding across model scales from 1.7B to 32B. Analyzing why standard matching fails out of distribution, we find it wins agreement with the teacher by draining mass from alternatives at high-entropy states, resulting in confident agreement on wrong answers, whereas GeoSD keeps those alternatives in reach.

Related papers

• OPD-Aha: From Linguistic Momentum to Visual Reflection in Multimodal On-Policy Distillation [15.986808086256792]
  Privileged on-policy distillation improves multimodal reasoning by allowing a teacher to evaluate student trajectories using rich, training-only visual evidence.<n>We find that the teacher's visual corrective preference is not lost under this misleading agreement.<n>We introduce OPD-Aha, which reconstructs the distillation target directly from this isolated visual preference.
  arXiv  Detail & Related papers  (2026-09-15T00:25:28Z)
• One Symptom, Three Levers: A Critical Review of On-Policy Self-Distillation [3.0867286767516937]
  On-policy distillation trains a language model on its own generations while a teacher scores them token by token.<n>The teacher is the model itself, conditioned on privileged information the student will not have at test time.<n>Early results were promising, with accuracy comparable to reinforcement learning at a fraction of the generated tokens.<n>But the same asymmetry that produces the signal also biases it.
  arXiv  Detail & Related papers  (2026-08-26T15:52:19Z)
• PAST: Privileged Adaptation from Complete Student Trajectories for On-Policy Self-Distillation [1.1440940588722308]
  On-policy self-distillation (OPSD) uses a privileged teacher to supervise a reasoning model on prefixes sampled from its own rollouts.<n>We introduce Privileged Adaptation from Student Trajectories (PAST)<n>PAST treats each completed student trajectory as additional privileged information for the OPSD teacher while leaving the student's distillation prefixes unchanged.
  arXiv  Detail & Related papers  (2026-08-09T14:20:12Z)
• DemoPSD: Disagreement-Modulated Policy Self-Distillation [49.30809363804017]
  On-policy self-distillation has emerged as a practical method for training large language models.<n>DemoPSD steers the student toward a weighted geometric combination of the teacher and student distributions.
  arXiv  Detail & Related papers  (2026-07-02T17:58:29Z)
• Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation [81.10000755917712]
  On-policy distillation transfers reasoning capabilities by training a student model on its own generated trajectories using token-level feedback from a teacher.<n>As student-generated prefixes lengthen, the teacher's next-token distribution becomes less confident and less discriminative.<n>To mitigate SFD, we introduce textbfLookahead Group Reward (ours).
  arXiv  Detail & Related papers  (2026-05-29T04:39:20Z)
• Prefix Teach, Suffix Fade: Local Teachability Collapse in Strong-to-Weak On-Policy Distillation [49.117085054884676]
  On-policy distillation trains a student model on its own rollouts using dense feedback from a stronger teacher.<n>We operationalize this principle through a trajectory-specific release rule.<n> Experimental results across strong-to-weak distillation tasks indicate that this release rule consistently outperforms standard full-trajectory OPD.
  arXiv  Detail & Related papers  (2026-05-13T15:05:30Z)
• Anti-Self-Distillation for Reasoning RL via Pointwise Mutual Information [22.436966302243565]
  We propose Anti-Self-Distillation, which ascends a divergence between student and teacher rather than descending it.<n>AntiSD reaches the GRPO baseline's accuracy in 2 to 10x fewer training steps and improves final accuracy by up to 11.5 points.
  arXiv  Detail & Related papers  (2026-05-12T06:40:43Z)
• Reasoning Compression with Mixed-Policy Distillation [20.7258653384411]
  Reasoning-centric large language models often incur excessive token usage and high inference-time decoding cost.<n>We propose Mixed-Policy Distillation (MPD), a reasoning compression framework that transfers concise reasoning behavior from a larger-sized teacher to a smaller student.<n>MPD reduces token usage by up to 27.1% while improving performance across multiple reasoning benchmarks.
  arXiv  Detail & Related papers  (2026-05-09T08:04:47Z)
• Prune-OPD: Efficient and Reliable On-Policy Distillation for Long-Horizon Reasoning [66.52232008796294]
  Prune-OPD dynamically aligns training budgets with supervision quality.<n>It reduces training time by 37.6%--68.0% while preserving, and often improving, performance on challenging benchmarks.
  arXiv  Detail & Related papers  (2026-05-08T14:38:53Z)
• Long-Chain Reasoning Distillation via Adaptive Prefix Alignment [57.130176131042965]
  We propose a framework that exploits teacher CoTs for distillation through adaptive prefix alignment.<n>P-ALIGN adaptively truncates teacher-generated reasoning trajectories by determining whether the remaining suffix is concise.<n>Experiments on multiple mathematical reasoning benchmarks demonstrate that P-ALIGN outperforms all baselines by over 3%.
  arXiv  Detail & Related papers  (2026-01-15T04:40:45Z)
• Warmup-Distill: Bridge the Distribution Mismatch between Teacher and Student before Knowledge Distillation [84.38105530043741]
  We propose Warmup-Distill, which aligns the distillation of the student to that of the teacher in advance of distillation.<n>Experiments on the seven benchmarks demonstrate that Warmup-Distill could provide a warmup student more suitable for distillation.
  arXiv  Detail & Related papers  (2025-02-17T12:58:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.