Distill What the Student Can See: Fisher-Projected On-Policy Distillation for Vision-Language Models

• URL: http://arxiv.org/abs/2608.01263v2
• Date: Thu, 06 Aug 2026 07:07:57 GMT
• Title: Distill What the Student Can See: Fisher-Projected On-Policy Distillation for Vision-Language Models
• Abstract summary: On-policy distillation (OPD) samples trajectories from the current student policy and minimizes token-level divergence between student and teacher next-token distributions.<n>We propose emphFisher-Projected On-Policy Distillation (FP-OPD), which distills only locally realizable teacher corrections.
• Score: 23.382646724389545
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: On-policy distillation (OPD) samples trajectories from the current student policy and minimizes token-level divergence between student and teacher next-token distributions at prefixes along those trajectories. This aligns the distillation states with the student's own generation distribution. However, it still assumes that the complete teacher distribution is an appropriate target across student capacities. In vision--language reasoning, teacher corrections can depend on visual distinctions that a compact student cannot represent. Our target-scaling study shows that, as the target approaches the complete teacher distribution, the student realizes less of the prescribed shift and obtains worse downstream performance. We therefore propose \emph{Fisher-Projected On-Policy Distillation} (FP-OPD), which distills only locally realizable teacher corrections. FP-OPD uses continuous visual perturbations to estimate the student's local visual tangent space and projects the centered teacher--student log-probability gap onto this space under the student's Fisher metric. The resulting capacity-aware target is optimized with full-vocabulary reverse KL on student trajectories, retaining the standard OPD framework. In 8B-to-2B distillation, FP-OPD improves all seven evaluated multimodal benchmarks. It raises the average score by 2.77 points over the pretrained student and by 1.60 points over standard OPD. These results demonstrate that locally realizable teacher corrections provide a more effective target for distilling compact vision--language models.

Related papers

• Distillation as Probability Transport: Routed On-Policy Distillation [29.500694918719166]
  RouteOPD (Routed On-Policy Distillation) decomposes local teacher--student disagreement into student-excess sources and teacher-deficit destinations.<n>Experiments across four teacher--student settings and four mathematical-reasoning benchmarks demonstrate that RouteOPD consistently outperforms sampled reverse-KL OPD.
  arXiv  Detail & Related papers  (2026-09-08T07:07:39Z)
• Learning to Track from Privileged Target Appearances [67.80976117600649]
  We quantify a bottleneck with a non-deployable oracle that supplies an exact current-frame target crop.<n>We introduce Privileged Appearance Transfer for Tracking (PATT), a teacher-student training framework that transfers these privileged appearances to a deployable tracker.<n>PATT achieves consistent gains under both long- and short-term tracking protocols.
  arXiv  Detail & Related papers  (2026-09-02T11:43:43Z)
• VISTA: Verifier-Informed Student-to-Teacher Adaptation for On-Policy Self-Distillation [19.198000863807316]
  On-policy self-distillation improves reasoning by training a problem-only student on its own rollouts using dense token-level supervision from a privileged teacher.<n>Standard OPSD treats the teacher distribution as a fixed target along the student's rollout and updates only the student %, although privileged conditioning does not guarantee that the teacher always provides the most appropriate target for problem-only reasoning.<n>We introduce Verifier-Informed Student-to-Teacher Adaptation (VISTA), which preserves the standard OPSD student update while using outcome-verified rollouts to adapt the teacher toward the student distribution.
  arXiv  Detail & Related papers  (2026-08-28T13:10:34Z)
• PAST: Privileged Adaptation from Complete Student Trajectories for On-Policy Self-Distillation [1.1440940588722308]
  On-policy self-distillation (OPSD) uses a privileged teacher to supervise a reasoning model on prefixes sampled from its own rollouts.<n>We introduce Privileged Adaptation from Student Trajectories (PAST)<n>PAST treats each completed student trajectory as additional privileged information for the OPSD teacher while leaving the student's distillation prefixes unchanged.
  arXiv  Detail & Related papers  (2026-08-09T14:20:12Z)
• STEP-OPD: Rethinking Output Targets and Internal Dynamics in On-Policy Distillation for Diffusion Models [14.682283642975271]
  We propose STEP-OPD, an on-policy distillation framework for image generation that extends the student's learning target beyond the teacher.<n>Instead of treating the teacher as the final target, we use the velocity difference between each task-specific teacher and the shared base model as a direction for further learning.<n>In addition, we align the direction and magnitude of representation changes between the student and teacher, enabling the student to learn how representations are progressively transformed.
  arXiv  Detail & Related papers  (2026-08-05T14:11:27Z)
• Weak-to-Strong On-Policy Distillation [21.651421588571296]
  We introduce Weak-to-Strong On-Policy Distillation (W2S-OPD), a framework that improves the strong student by distilling from multiple weak models.<n>Across four math and three code benchmarks, W2S-OPD outperforms OPD, enables the student to surpass the domain teacher, and keeps improving the student even when every supervision source is weaker.
  arXiv  Detail & Related papers  (2026-07-28T20:31:48Z)
• Contrastive On-Policy Distillation [36.47487086937924]
  We propose COPD, a contrastive On-policy Distillation framework.<n> COPD directly encourages the student model to learn more concise and efficient reasoning strategies.
  arXiv  Detail & Related papers  (2026-07-21T12:35:04Z)
• DemoPSD: Disagreement-Modulated Policy Self-Distillation [49.30809363804017]
  On-policy self-distillation has emerged as a practical method for training large language models.<n>DemoPSD steers the student toward a weighted geometric combination of the teacher and student distributions.
  arXiv  Detail & Related papers  (2026-07-02T17:58:29Z)
• DOPD: Dual On-policy Distillation [59.65986569617147]
  On-policy distillation offers superior capacity transfer by supervising student-sampled trajectories with dense token-level signals.<n>We propose DOPD, an advantage-aware dual distillation paradigm that dynamically routes token-level supervision between privileged teacher and privileged student policies.<n>Experiments on both large language model (LLM) and vision-language model (VLM) settings demonstrate that DOPD consistently outperforms Vanilla OPD.
  arXiv  Detail & Related papers  (2026-06-29T17:55:53Z)
• On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agents [50.78688247330029]
  On-Policy Distillation (OPD) is a natural recipe for transferring such capabilities to smaller students.<n>We propose Guided On-Policy Distillation (Guided-OPD), a simple yet effective algorithm that mixes teacher- and student-generated turns within each rollout.<n>Guided-OPD improves Score by 21.1% and Success Rate by 25.5% over vanilla OPD on average.
  arXiv  Detail & Related papers  (2026-06-14T16:41:45Z)
• When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoning [45.79647925282674]
  On-policy self-distillation (OPSD) trains a student on its own rollouts using a privileged teacher.<n>Existing entropy-based OPD methods relax this uniformity by modulating token-level supervision with teacher entropy.<n>We propose Position-Weighted On-Policy Self-Distillation (PW-OPSD), which applies an increasing position weight while keeping the same student rollout, privileged teacher pass, and clipped forward-KL target as OPSD.
  arXiv  Detail & Related papers  (2026-05-20T18:14:03Z)
• Prefix Teach, Suffix Fade: Local Teachability Collapse in Strong-to-Weak On-Policy Distillation [49.117085054884676]
  On-policy distillation trains a student model on its own rollouts using dense feedback from a stronger teacher.<n>We operationalize this principle through a trajectory-specific release rule.<n> Experimental results across strong-to-weak distillation tasks indicate that this release rule consistently outperforms standard full-trajectory OPD.
  arXiv  Detail & Related papers  (2026-05-13T15:05:30Z)
• On-Policy Distillation with Best-of-N Teacher Rollout Selection [54.91780727674628]
  We propose BRTS, a Best-of-N Rollout Teacher Selection framework for on-policy distillation.<n>BRTS augments standard student-context OPD with a teacher-context supervision branch constructed from the teacher trajectory.<n>BRTS improves over standard OPD on challenging reasoning benchmarks, with the largest gains on harder datasets.
  arXiv  Detail & Related papers  (2026-05-10T19:49:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.