Boosting Contrastive Learning with Relation Knowledge Distillation
- URL: http://arxiv.org/abs/2112.04174v1
- Date: Wed, 8 Dec 2021 08:49:18 GMT
- Title: Boosting Contrastive Learning with Relation Knowledge Distillation
- Authors: Kai Zheng, Yuanjiang Wang, Ye Yuan
- Abstract summary: We propose a relation-wise contrastive paradigm with Relation Knowledge Distillation (ReKD)
We show that our method achieves significant improvements on multiple lightweight models.
- Score: 12.14219750487548
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: While self-supervised representation learning (SSL) has proved to be
effective in the large model, there is still a huge gap between the SSL and
supervised method in the lightweight model when following the same solution. We
delve into this problem and find that the lightweight model is prone to
collapse in semantic space when simply performing instance-wise contrast. To
address this issue, we propose a relation-wise contrastive paradigm with
Relation Knowledge Distillation (ReKD). We introduce a heterogeneous teacher to
explicitly mine the semantic information and transferring a novel relation
knowledge to the student (lightweight model). The theoretical analysis supports
our main concern about instance-wise contrast and verify the effectiveness of
our relation-wise contrastive learning. Extensive experimental results also
demonstrate that our method achieves significant improvements on multiple
lightweight models. Particularly, the linear evaluation on AlexNet obviously
improves the current state-of-art from 44.7% to 50.1%, which is the first work
to get close to the supervised 50.5%. Code will be made available.
Related papers
- Retro: Reusing teacher projection head for efficient embedding distillation on Lightweight Models via Self-supervised Learning [0.0]
We propose textscRetro, which reuses the teacher's projection head for students.
Our experimental results demonstrate significant improvements over the state-of-the-art on all lightweight models.
arXiv Detail & Related papers (2024-05-24T07:53:09Z) - A Probabilistic Model Behind Self-Supervised Learning [53.64989127914936]
In self-supervised learning (SSL), representations are learned via an auxiliary task without annotated labels.
We present a generative latent variable model for self-supervised learning.
We show that several families of discriminative SSL, including contrastive methods, induce a comparable distribution over representations.
arXiv Detail & Related papers (2024-02-02T13:31:17Z) - ArCL: Enhancing Contrastive Learning with Augmentation-Robust
Representations [30.745749133759304]
We develop a theoretical framework to analyze the transferability of self-supervised contrastive learning.
We show that contrastive learning fails to learn domain-invariant features, which limits its transferability.
Based on these theoretical insights, we propose a novel method called Augmentation-robust Contrastive Learning (ArCL)
arXiv Detail & Related papers (2023-03-02T09:26:20Z) - On Higher Adversarial Susceptibility of Contrastive Self-Supervised
Learning [104.00264962878956]
Contrastive self-supervised learning (CSL) has managed to match or surpass the performance of supervised learning in image and video classification.
It is still largely unknown if the nature of the representation induced by the two learning paradigms is similar.
We identify the uniform distribution of data representation over a unit hypersphere in the CSL representation space as the key contributor to this phenomenon.
We devise strategies that are simple, yet effective in improving model robustness with CSL training.
arXiv Detail & Related papers (2022-07-22T03:49:50Z) - Weak Augmentation Guided Relational Self-Supervised Learning [80.0680103295137]
We introduce a novel relational self-supervised learning (ReSSL) framework that learns representations by modeling the relationship between different instances.
Our proposed method employs sharpened distribution of pairwise similarities among different instances as textitrelation metric.
Experimental results show that our proposed ReSSL substantially outperforms the state-of-the-art methods across different network architectures.
arXiv Detail & Related papers (2022-03-16T16:14:19Z) - ReSSL: Relational Self-Supervised Learning with Weak Augmentation [68.47096022526927]
Self-supervised learning has achieved great success in learning visual representations without data annotations.
We introduce a novel relational SSL paradigm that learns representations by modeling the relationship between different instances.
Our proposed ReSSL significantly outperforms the previous state-of-the-art algorithms in terms of both performance and training efficiency.
arXiv Detail & Related papers (2021-07-20T06:53:07Z) - DisCo: Remedy Self-supervised Learning on Lightweight Models with
Distilled Contrastive Learning [94.89221799550593]
Self-supervised representation learning (SSL) has received widespread attention from the community.
Recent research argue that its performance will suffer a cliff fall when the model size decreases.
We propose a simple yet effective Distilled Contrastive Learning (DisCo) to ease the issue by a large margin.
arXiv Detail & Related papers (2021-04-19T08:22:52Z) - SEED: Self-supervised Distillation For Visual Representation [34.63488756535054]
We propose a new learning paradigm, named SElf-SupErvised Distillation (SEED), to transfer its representational knowledge into a smaller architecture (as Student) in a self-supervised fashion.
We show that SEED dramatically boosts the performance of small networks on downstream tasks.
arXiv Detail & Related papers (2021-01-12T20:04:50Z) - Relation-Guided Representation Learning [53.60351496449232]
We propose a new representation learning method that explicitly models and leverages sample relations.
Our framework well preserves the relations between samples.
By seeking to embed samples into subspace, we show that our method can address the large-scale and out-of-sample problem.
arXiv Detail & Related papers (2020-07-11T10:57:45Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.