A Study of the Generalizability of Self-Supervised Representations
- URL: http://arxiv.org/abs/2109.09150v1
- Date: Sun, 19 Sep 2021 15:57:37 GMT
- Title: A Study of the Generalizability of Self-Supervised Representations
- Authors: Atharva Tendle and Mohammad Rashedul Hasan
- Abstract summary: Recent advancements in self-supervised learning (SSL) made it possible to learn generalizable visual representations from unlabeled data.
We study generalizability of the SSL and SL-based models via their prediction accuracy as well as prediction confidence.
We show that the SSL representations are more generalizable as compared to the SL representations.
- Score: 0.0
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Recent advancements in self-supervised learning (SSL) made it possible to
learn generalizable visual representations from unlabeled data. The performance
of Deep Learning models fine-tuned on pretrained SSL representations is on par
with models fine-tuned on the state-of-the-art supervised learning (SL)
representations. Irrespective of the progress made in SSL, its generalizability
has not been studied extensively. In this article, we perform a deeper analysis
of the generalizability of pretrained SSL and SL representations by conducting
a domain-based study for transfer learning classification tasks. The
representations are learned from the ImageNet source data, which are then
fine-tuned using two types of target datasets: similar to the source dataset,
and significantly different from the source dataset. We study generalizability
of the SSL and SL-based models via their prediction accuracy as well as
prediction confidence. In addition to this, we analyze the attribution of the
final convolutional layer of these models to understand how they reason about
the semantic identity of the data. We show that the SSL representations are
more generalizable as compared to the SL representations. We explain the
generalizability of the SSL representations by investigating its invariance
property, which is shown to be better than that observed in the SL
representations.
Related papers
- A Closer Look at Benchmarking Self-Supervised Pre-training with Image Classification [51.35500308126506]
Self-supervised learning (SSL) is a machine learning approach where the data itself provides supervision, eliminating the need for external labels.
We study how classification-based evaluation protocols for SSL correlate and how well they predict downstream performance on different dataset types.
arXiv Detail & Related papers (2024-07-16T23:17:36Z) - Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects [84.6945070729684]
Self-supervised learning (SSL) has recently achieved impressive performance on various time series tasks.
This article reviews current state-of-the-art SSL methods for time series data.
arXiv Detail & Related papers (2023-06-16T18:23:10Z) - Reverse Engineering Self-Supervised Learning [17.720366509919167]
Self-supervised learning (SSL) is a powerful tool in machine learning.
This paper presents an in-depth empirical analysis of SSL-trained representations.
arXiv Detail & Related papers (2023-05-24T23:15:28Z) - Explaining, Analyzing, and Probing Representations of Self-Supervised
Learning Models for Sensor-based Human Activity Recognition [2.2082422928825136]
Self-supervised learning (SSL) frameworks have been extensively applied to sensor-based Human Activity Recognition (HAR)
In this paper, we aim to analyze deep representations of two recent SSL frameworks, namely SimCLR and VICReg.
arXiv Detail & Related papers (2023-04-14T07:53:59Z) - Evaluating Self-Supervised Learning via Risk Decomposition [100.73914689472507]
Self-supervised learning (SSL) pipelines differ in many design choices such as the architecture, augmentations, or pretraining data.
This does not provide much insight into why or when a model is better, now how to improve it.
We propose an SSL risk decomposition, which generalizes the classical supervised approximation-estimation decomposition.
arXiv Detail & Related papers (2023-02-06T19:09:00Z) - Self-Supervised PPG Representation Learning Shows High Inter-Subject
Variability [3.8036939971290007]
We propose a Self-Supervised Learning (SSL) method with a pretext task of signal reconstruction to learn an informative generalized PPG representation.
Results show that in a very limited label data setting (10 samples per class or less), using SSL is beneficial.
SSL may pave the way for the broader use of machine learning models on PPG data in label-scarce regimes.
arXiv Detail & Related papers (2022-12-07T19:02:45Z) - Improving Self-Supervised Learning by Characterizing Idealized
Representations [155.1457170539049]
We prove necessary and sufficient conditions for any task invariant to given data augmentations.
For contrastive learning, our framework prescribes simple but significant improvements to previous methods.
For non-contrastive learning, we use our framework to derive a simple and novel objective.
arXiv Detail & Related papers (2022-09-13T18:01:03Z) - High Fidelity Visualization of What Your Self-Supervised Representation
Knows About [22.982471878833362]
In this work, we showcase the use of a conditional diffusion based generative model (RCDM) to visualize representations learned with self-supervised models.
We demonstrate how this model's generation quality is on par with state-of-the-art generative models while being faithful to the representation used as conditioning.
arXiv Detail & Related papers (2021-12-16T19:23:33Z) - Self-supervised Learning is More Robust to Dataset Imbalance [65.84339596595383]
We investigate self-supervised learning under dataset imbalance.
Off-the-shelf self-supervised representations are already more robust to class imbalance than supervised representations.
We devise a re-weighted regularization technique that consistently improves the SSL representation quality on imbalanced datasets.
arXiv Detail & Related papers (2021-10-11T06:29:56Z) - Graph-based Semi-supervised Learning: A Comprehensive Review [51.26862262550445]
Semi-supervised learning (SSL) has tremendous value in practice due to its ability to utilize both labeled data and unlabelled data.
An important class of SSL methods is to naturally represent data as graphs, which corresponds to graph-based semi-supervised learning (GSSL) methods.
GSSL methods have demonstrated their advantages in various domains due to their uniqueness of structure, the universality of applications, and their scalability to large scale data.
arXiv Detail & Related papers (2021-02-26T05:11:09Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.