CODI: Improving Self-Supervised Vision Transformers with Cross Distillation
Abstract Overview
This paper examines why self-supervised Vision Transformers can produce strong image-level representations while still yielding weaker patch-level features for dense prediction. The authors show empirically that the [CLS] token tends to concentrate on a small set of representative patches, and that dense prediction errors are more common on low-attention patches. They provide a local gradient analysis indicating that semantic gradients from image-level self-distillation are weighted by [CLS]-to-patch attention mass, which creates uneven semantic supervision across patches. To address this, they propose Cross Distillation (CODI), which mixes a small amount of the teacher class-token distribution into each teacher patch-token target without changing the model architecture.
Novelty
The distinctive contribution is a target-side cross-distillation method that injects global class-token semantics directly into masked patch supervision, rather than modifying the backbone or adding a separate dense-training stage. The paper also links the dense-representation weakness to a representative-patch phenomenon through both empirical evidence and a gradient-based analysis of the [CLS] attention path.
Results
Under aligned ImageNet-1K pretraining settings, CODI improves both dense and image-level transfer over reproduced DINOv2 across multiple model sizes. For semantic segmentation, the paper reports average mIoU gains of +2.7, +1.7, and +1.9 for ViT-S/14, ViT-B/14, and ViT-L/14 respectively, and it matches or improves DINOv2 in 24 of 27 depth-estimation metric comparisons. CODI also reaches 66.0 J&F on DAVIS with ViT-L/14 and shows consistent classification and occlusion-robustness gains.
Key Points
- The paper identifies a mismatch between image-level and patch-level learning in self-supervised ViTs: [CLS] attention emphasizes a few representative patches, while dense errors concentrate on low-attention regions.
- CODI forms each masked patch target by mixing the teacher patch distribution with a small amount of the teacher class-token distribution, providing attention-independent global semantic guidance to every masked patch.
- Experiments across segmentation, depth estimation, video object segmentation, and classification indicate that CODI improves dense transfer while also preserving or slightly improving image-level performance under comparable pretraining settings.