Visual Contrastive Self-Distillation

• URL: http://arxiv.org/abs/2607.21556v1
• Date: Thu, 23 Jul 2026 17:37:37 GMT
• Title: Visual Contrastive Self-Distillation
• Abstract summary: We propose Visual Contrastive Self-Distillation, which converts image-content removal into an on-policy self-distillation signal.<n>We use this contrast to sharpen the teacher's original-image distribution within its plausible support, and distill the resulting full-distribution target into the student.
• Score: 61.298159957622886
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: On-policy self-distillation (OPSD) is promising as it removes the external teacher required by on-policy distillation (OPD), yet it still needs asymmetric information between teacher and student to ensure that the self-teacher provides a stronger learning signal than the student. Existing methods create this asymmetry either through privileged answers or visual evidence. We ask whether both can be removed, yielding a simpler form of OPSD driven purely by input conditioning. For this purpose, we propose Visual Contrastive Self-Distillation, namely VCSD, which converts image-content removal into an on-policy self-distillation signal. At each student-generated response prefix, the EMA teacher produces two next-token distributions under the same prompt and prefix -- one conditioned on the original image and the other on a content-erased control. Their token-wise log-probability difference highlights candidates whose likelihood is specifically increased by the instance-level visual content. We use this contrast to sharpen the teacher's original-image distribution within its plausible support, and distill the resulting full-distribution target into the student. Using ViRL39K dataset, VCSD consistently outperforms matched OPSD across Qwen3-VL and Qwen3.5 models. For example, on Qwen3-VL, it improves the seven-benchmark aggregate from $62.27\% \rightarrow 67.04\%$ at 2B, $71.30\% \rightarrow 73.16\%$ at 4B, and $72.51\% \rightarrow 76.26\%$ at 8B. Furthermore, VCSD requires no external teacher, privileged answers, visual evidence signals, reasoning traces, or additional inference-time cost.

Related papers

• Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement [27.122890248991556]
  On-policy distillation (OPD) offers dense token-level supervision as an alternative to the sparse outcome-level advantages of reinforcement learning with verifiable rewards (RLVR)<n>We quantitatively analyze teacher supervision during OPD training and find substantial noise whose prevalence increases with teacher scale.<n>We find that learning concentrates on low log-probability tokens, and using a single fixed negative advantage matches the performance of teacher-provided ones.
  arXiv  Detail & Related papers  (2026-08-31T16:22:10Z)
• Self-Supervised Visual On-Policy Distillation [63.83705343579541]
  Visual on-policy distillation relies heavily on an informative teacher-student asymmetry.<n>We introduce Self-Supervised Visual On-Policy Distillation (S$2$VOPD)<n>S$2$VOPD distills the teacher's distribution conditioned on the original image on-policy into the student distribution conditioned on a strongly augmented view of the same image.
  arXiv  Detail & Related papers  (2026-08-14T09:52:01Z)
• H$^2$SD: Hybrid Hindsight Self-Distillation [61.71131241473028]
  Hybrid Hindsight Self-Distillation ($mathrmH2mathrmSD$) adapts teacher context and update strategy to trajectory correctness.<n>For successful trajectories, we construct the teacher context from the verified response and a rephrasing instruction.<n>For failed trajectories, a verifier-confirmed reference hint provides corrective guidance through reverse-KL distillation.
  arXiv  Detail & Related papers  (2026-07-21T10:47:27Z)
• DemoPSD: Disagreement-Modulated Policy Self-Distillation [49.30809363804017]
  On-policy self-distillation has emerged as a practical method for training large language models.<n>DemoPSD steers the student toward a weighted geometric combination of the teacher and student distributions.
  arXiv  Detail & Related papers  (2026-07-02T17:58:29Z)
• DOPD: Dual On-policy Distillation [59.65986569617147]
  On-policy distillation offers superior capacity transfer by supervising student-sampled trajectories with dense token-level signals.<n>We propose DOPD, an advantage-aware dual distillation paradigm that dynamically routes token-level supervision between privileged teacher and privileged student policies.<n>Experiments on both large language model (LLM) and vision-language model (VLM) settings demonstrate that DOPD consistently outperforms Vanilla OPD.
  arXiv  Detail & Related papers  (2026-06-29T17:55:53Z)
• Visual-OPSD: Cross-Modal On-Policy Self-Distillation for Efficient Unified Multimodal Reasoning [30.851590309402436]
  Unified multimodal models (UMMs) interleave generated ''visual thoughts'' (VTs) with text reasoning to improve spatial tasks.<n>This incurs roughly an order-of-magnitude inference cost from multi-step diffusion.<n>We propose Visual On-Policy Self-Distillation(Visual-OPSD) to solve this problem.
  arXiv  Detail & Related papers  (2026-06-17T11:59:15Z)
• ViCuR: Visual Cues as Recoverable Privilege for Multimodal On-Policy Distillation [5.131753844398592]
  ViCuR is a visually grounded privileged-teacher distillation framework.<n>It replaces answer-side privilege with visual cues.<n>It consistently improves over answer-based on-policy self-distillation.
  arXiv  Detail & Related papers  (2026-06-04T05:18:13Z)
• Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation [81.10000755917712]
  On-policy distillation transfers reasoning capabilities by training a student model on its own generated trajectories using token-level feedback from a teacher.<n>As student-generated prefixes lengthen, the teacher's next-token distribution becomes less confident and less discriminative.<n>To mitigate SFD, we introduce textbfLookahead Group Reward (ours).
  arXiv  Detail & Related papers  (2026-05-29T04:39:20Z)
• AVSD: Adaptive-View Self-Distillation by Balancing Consensus and Teacher-Specific Privileged Signals [65.71225905458182]
  Self-distillation enables language models to learn on-policy from their own trajectories by using the same model as both student and teacher.<n>This setup provides dense token-level feedback without relying on a separate external model.<n>We introduce AVSD, a novel method of self-distillation with multiple privileged-information views.
  arXiv  Detail & Related papers  (2026-05-20T03:06:36Z)
• Adaptive Teacher Exposure for Self-Distillation in LLM Reasoning [2.9029480541171107]
  We propose adaptive teacher exposure as an effective new axis for reasoning self-distillation.<n>We show that Adaptive Teacher Exposure for Self-Distillation (ATESD) consistently outperforms competitive self-distillation and RL baselines.
  arXiv  Detail & Related papers  (2026-05-12T03:15:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.