Evaluating Self-Supervised Learning via Risk Decomposition
- URL: http://arxiv.org/abs/2302.03068v3
- Date: Mon, 8 Jan 2024 05:04:44 GMT
- Title: Evaluating Self-Supervised Learning via Risk Decomposition
- Authors: Yann Dubois and Tatsunori Hashimoto and Percy Liang
- Abstract summary: Self-supervised learning (SSL) pipelines differ in many design choices such as the architecture, augmentations, or pretraining data.
This does not provide much insight into why or when a model is better, now how to improve it.
We propose an SSL risk decomposition, which generalizes the classical supervised approximation-estimation decomposition.
- Score: 100.73914689472507
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Self-supervised learning (SSL) pipelines differ in many design choices such
as the architecture, augmentations, or pretraining data. Yet SSL is typically
evaluated using a single metric: linear probing on ImageNet. This does not
provide much insight into why or when a model is better, now how to improve it.
To address this, we propose an SSL risk decomposition, which generalizes the
classical supervised approximation-estimation decomposition by considering
errors arising from the representation learning step. Our decomposition
consists of four error components: approximation, representation usability,
probe generalization, and encoder generalization. We provide efficient
estimators for each component and use them to analyze the effect of 30 design
choices on 169 SSL vision models evaluated on ImageNet. Our analysis gives
valuable insights for designing and using SSL models. For example, it
highlights the main sources of error and shows how to improve SSL in specific
settings (full- vs few-shot) by trading off error components. All results and
pretrained models are at https://github.com/YannDubs/SSL-Risk-Decomposition.
Related papers
- A Closer Look at Benchmarking Self-Supervised Pre-training with Image Classification [51.35500308126506]
Self-supervised learning (SSL) is a machine learning approach where the data itself provides supervision, eliminating the need for external labels.
We study how classification-based evaluation protocols for SSL correlate and how well they predict downstream performance on different dataset types.
arXiv Detail & Related papers (2024-07-16T23:17:36Z) - FroSSL: Frobenius Norm Minimization for Efficient Multiview Self-Supervised Learning [8.572896815776089]
FroSSL reconciles covariance eigenvalue regularization and using more views.
We show that FroSSL reaches competitive accuracies more quickly than any other SSL method.
We also show that FroSSL learns competitive representations on linear probe evaluation when used to train a ResNet-18 on several datasets.
arXiv Detail & Related papers (2023-10-04T15:42:23Z) - Evaluating The Robustness of Self-Supervised Representations to
Background/Foreground Removal [4.007351600492541]
We consider state-of-the-art SSL pretrained models, such as DINOv2, MAE, and SwaV, and analyzed changes at the representation levels across 4 Image Classification datasets.
Empirically, we show that not all models lead to representations that separate foreground, background, and complete images.
arXiv Detail & Related papers (2023-06-02T09:46:22Z) - Benchmark for Uncertainty & Robustness in Self-Supervised Learning [0.0]
Self-Supervised Learning is crucial for real-world applications, especially in data-hungry domains such as healthcare and self-driving cars.
In this paper, we explore variants of SSL methods, including Jigsaw Puzzles, Context, Rotation, Geometric Transformations Prediction for vision, as well as BERT and GPT for language tasks.
Our goal is to create a benchmark with outputs from experiments, providing a starting point for new SSL methods in Reliable Machine Learning.
arXiv Detail & Related papers (2022-12-23T15:46:23Z) - Improving Self-Supervised Learning by Characterizing Idealized
Representations [155.1457170539049]
We prove necessary and sufficient conditions for any task invariant to given data augmentations.
For contrastive learning, our framework prescribes simple but significant improvements to previous methods.
For non-contrastive learning, we use our framework to derive a simple and novel objective.
arXiv Detail & Related papers (2022-09-13T18:01:03Z) - Self-supervised Learning is More Robust to Dataset Imbalance [65.84339596595383]
We investigate self-supervised learning under dataset imbalance.
Off-the-shelf self-supervised representations are already more robust to class imbalance than supervised representations.
We devise a re-weighted regularization technique that consistently improves the SSL representation quality on imbalanced datasets.
arXiv Detail & Related papers (2021-10-11T06:29:56Z) - Rethinking Self-Supervised Learning: Small is Beautiful [30.809693803413445]
We propose scaled-down self-supervised learning (S3L), which include 3 parts: small resolution, small architecture and small data.
On a diverse set of datasets, S3L achieves higher accuracy consistently with much less training cost when compared to previous SSL learning paradigm.
arXiv Detail & Related papers (2021-03-25T01:48:52Z) - Understanding self-supervised Learning Dynamics without Contrastive
Pairs [72.1743263777693]
Contrastive approaches to self-supervised learning (SSL) learn representations by minimizing the distance between two augmented views of the same data point.
BYOL and SimSiam, show remarkable performance it without negative pairs.
We study the nonlinear learning dynamics of non-contrastive SSL in simple linear networks.
arXiv Detail & Related papers (2021-02-12T22:57:28Z) - Interventional Few-Shot Learning [88.31112565383457]
We propose a novel Few-Shot Learning paradigm: Interventional Few-Shot Learning.
Code is released at https://github.com/yue-zhongqi/ifsl.
arXiv Detail & Related papers (2020-09-28T01:16:54Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.