Daily Paper Briefing

Abstract Overview

This paper introduces Visual Contrastive Self-Distillation (VCSD), an on-policy self-distillation method for vision-language models that does not rely on an external teacher, privileged answers, reasoning traces, or explicit visual evidence signals. At each student-generated prefix, an EMA teacher is evaluated twice under the same prompt and prefix: once with the original image and once with a content-erased control image. The token-level log-probability difference between these two conditions is used to sharpen the teacher’s original-image distribution within a plausibility-constrained support, and the resulting full distribution is distilled into the student with forward KL. The method is evaluated on ViRL39K using Qwen3-VL and Qwen3.5 models across seven vision-language benchmarks.

Novelty

The distinctive idea is to create the teacher-student asymmetry needed for on-policy self-distillation purely through matched input conditioning, rather than through privileged supervision or evidence-focused image processing. VCSD uses contrast between teacher predictions under the real image and a content-erased control to form a visually informed full-distribution target anchored by the original-image distribution.

Results

Across six model configurations, VCSD consistently outperforms both the corresponding base models and matched OPSD baselines on the seven-benchmark aggregate. For Qwen3-VL, aggregate accuracy improves from 62.27% to 67.04% at 2B, 71.30% to 73.16% at 4B, and 72.51% to 76.26% at 8B; for Qwen3.5 models, gains over base models range from 2.90% to 4.27%. Ablations further indicate that plausibility restriction stabilizes training, moderate contrast strength works best, and forward KL is the strongest distillation divergence among those tested.

Key Points

1. VCSD contrasts EMA-teacher next-token distributions under the original image and a content-erased control, then distills a plausibility-constrained, contrast-shaped target into the student.
2. The method removes the need for external teachers and auxiliary supervision such as privileged answers, reasoning traces, evidence crops, or extra inference-time branches.
3. Experiments on ViRL39K with Qwen3-VL and Qwen3.5 show consistent aggregate gains over both base models and matched OPSD across seven vision-language benchmarks.

References

• arXiv: https://arxiv.org/abs/2607.21556v1
• Fugu-MT: https://fugumt.com/fugumt/paper_check/2607.21556v1
• Hugging Face Papers: https://huggingface.co/papers/2607.21556