DinoTwins: Combining DINO and Barlow Twins for Robust, Label-Efficient Vision Transformers

• URL: http://arxiv.org/abs/2508.17509v1
• Date: Sun, 24 Aug 2025 20:18:05 GMT
• Title: DinoTwins: Combining DINO and Barlow Twins for Robust, Label-Efficient Vision Transformers
• Authors: Michael Podsiadly, Brendon K Lay,
• Abstract summary: We combine DINO (teacher-student learning) and Barlow Twins (redundancy reduction) to create a model that learns better with fewer labels and less compute.<n>Preliminary results show that the combined approach achieves comparable loss and classification accuracy to DINO while maintaining strong feature representations.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Training AI models to understand images without costly labeled data remains a challenge. We combine two techniques--DINO (teacher-student learning) and Barlow Twins (redundancy reduction)--to create a model that learns better with fewer labels and less compute. While both DINO and Barlow Twins have independently demonstrated strong performance in self-supervised learning, each comes with limitations--DINO may be sensitive to certain augmentations, and Barlow Twins often requires batch sizes too large to fit on consumer hardware. By combining the redundancy-reduction objective of Barlow Twins with the self-distillation strategy of DINO, we aim to leverage their complementary strengths. We train a hybrid model on the MS COCO dataset using only 10\% of labeled data for linear probing, and evaluate its performance against standalone DINO and Barlow Twins implementations. Preliminary results show that the combined approach achieves comparable loss and classification accuracy to DINO while maintaining strong feature representations. Attention visualizations further suggest improved semantic segmentation capability in the hybrid model. This combined method offers a scalable, label-efficient alternative for training ViTs in resource-constrained environments.

Related papers

• Dual-granularity Sinkhorn Distillation for Enhanced Learning from Long-tailed Noisy Data [67.25796812343454]
  Real-world datasets for deep learning frequently suffer from the co-occurring challenges of class imbalance and label noise.<n>We propose Dual-granularity Sinkhorn Distillation (D-SINK), a novel framework that enhances dual robustness by distilling and integrating complementary insights.<n>Experiments on benchmark datasets demonstrate that D-SINK significantly improves robustness and achieves strong empirical performance in learning from long-tailed noisy data.
  arXiv  Detail & Related papers  (2025-10-09T13:05:27Z)
• Mono2Stereo: Monocular Knowledge Transfer for Enhanced Stereo Matching [7.840781070208874]
  We propose leveraging monocular knowledge transfer to enhance stereo matching, namely Mono2Stereo. We introduce knowledge transfer with a two-stage training process, comprising synthetic data pre-training and real-world data fine-tuning. Experimental results demonstrate that our pre-trained model exhibits strong zero-shot capabilities.
  arXiv  Detail & Related papers  (2024-11-14T03:01:36Z)
• Robust Training of Federated Models with Extremely Label Deficiency [84.00832527512148]
  Federated semi-supervised learning (FSSL) has emerged as a powerful paradigm for collaboratively training machine learning models using distributed data with label deficiency. We propose a novel twin-model paradigm, called Twin-sight, designed to enhance mutual guidance by providing insights from different perspectives of labeled and unlabeled data. Our comprehensive experiments on four benchmark datasets provide substantial evidence that Twin-sight can significantly outperform state-of-the-art methods across various experimental settings.
  arXiv  Detail & Related papers  (2024-02-22T10:19:34Z)
• Guarding Barlow Twins Against Overfitting with Mixed Samples [27.7244906436942]
  Self-supervised learning aims to learn transferable feature representations for downstream applications without relying on labeled data. We introduce Mixed Barlow Twins, which aims to improve sample interaction during Barlow Twins training via linearly interpolated samples.
  arXiv  Detail & Related papers  (2023-12-04T18:59:36Z)
• BTSeg: Barlow Twins Regularization for Domain Adaptation in Semantic Segmentation [3.5229503563299915]
  We introduce BTSeg, an innovative, semi-supervised training approach enhancing semantic segmentation models. Images captured at similar locations but under varying adverse conditions are regarded as manifold representation of the same scene, thereby enabling the model to conceptualize its understanding of the environment.
  arXiv  Detail & Related papers  (2023-08-31T15:49:53Z)
• Deep Active Learning Using Barlow Twins [0.0]
  The generalisation performance of a convolutional neural networks (CNN) is majorly predisposed by the quantity, quality, and diversity of the training images. The goal of the Active learning for the task is to draw most informative samples from the unlabeled pool. We propose Deep Active Learning using BarlowTwins(DALBT), an active learning method for all the datasets.
  arXiv  Detail & Related papers  (2022-12-30T12:39:55Z)
• Non-contrastive representation learning for intervals from well logs [58.70164460091879]
  The representation learning problem in the oil & gas industry aims to construct a model that provides a representation based on logging data for a well interval. One of the possible approaches is self-supervised learning (SSL) We are the first to introduce non-contrastive SSL for well-logging data.
  arXiv  Detail & Related papers  (2022-09-28T13:27:10Z)
• Adversarial Dual-Student with Differentiable Spatial Warping for Semi-Supervised Semantic Segmentation [70.2166826794421]
  We propose a differentiable geometric warping to conduct unsupervised data augmentation. We also propose a novel adversarial dual-student framework to improve the Mean-Teacher. Our solution significantly improves the performance and state-of-the-art results are achieved on both datasets.
  arXiv  Detail & Related papers  (2022-03-05T17:36:17Z)
• L2B: Learning to Bootstrap Robust Models for Combating Label Noise [52.02335367411447]
  This paper introduces a simple and effective method, named Learning to Bootstrap (L2B) It enables models to bootstrap themselves using their own predictions without being adversely affected by erroneous pseudo-labels. It achieves this by dynamically adjusting the importance weight between real observed and generated labels, as well as between different samples through meta-learning.
  arXiv  Detail & Related papers  (2022-02-09T05:57:08Z)
• Self-Damaging Contrastive Learning [92.34124578823977]
  Unlabeled data in reality is commonly imbalanced and shows a long-tail distribution. This paper proposes a principled framework called Self-Damaging Contrastive Learning to automatically balance the representation learning without knowing the classes. Our experiments show that SDCLR significantly improves not only overall accuracies but also balancedness.
  arXiv  Detail & Related papers  (2021-06-06T00:04:49Z)
• Graph Barlow Twins: A self-supervised representation learning framework for graphs [25.546290138565393]
  We propose a framework for self-supervised graph representation learning - Graph Barlow Twins. It utilizes a cross-correlation-based loss function instead of negative samples. We show that our method achieves as competitive results as the best self-supervised methods and fully supervised ones.
  arXiv  Detail & Related papers  (2021-06-04T13:10:51Z)
• Barlow Twins: Self-Supervised Learning via Redundancy Reduction [31.077182488826963]
  Self-supervised learning (SSL) is rapidly closing the gap with supervised methods on large computer vision benchmarks. We propose an objective function that naturally avoids collapse by measuring the cross-correlation matrix between the outputs of two identical networks. This causes the representation vectors of distorted versions of a sample to be similar, while minimizing the redundancy between the components of these vectors.
  arXiv  Detail & Related papers  (2021-03-04T18:55:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.