Whitening Consistently Improves Self-Supervised Learning

• URL: http://arxiv.org/abs/2408.07519v1
• Date: Wed, 14 Aug 2024 12:52:13 GMT
• Title: Whitening Consistently Improves Self-Supervised Learning
• Authors: András Kalapos, Bálint Gyires-Tóth,
• Abstract summary: We propose incorporating ZCA whitening as the final layer of the encoder in self-supervised learning. Our experiments show that whitening is capable of improving linear and k-NN probing accuracy by 1-5%.
• Score: 5.0337106694127725
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Self-supervised learning (SSL) has been shown to be a powerful approach for learning visual representations. In this study, we propose incorporating ZCA whitening as the final layer of the encoder in self-supervised learning to enhance the quality of learned features by normalizing and decorrelating them. Although whitening has been utilized in SSL in previous works, its potential to universally improve any SSL model has not been explored. We demonstrate that adding whitening as the last layer of SSL pretrained encoders is independent of the self-supervised learning method and encoder architecture, thus it improves performance for a wide range of SSL methods across multiple encoder architectures and datasets. Our experiments show that whitening is capable of improving linear and k-NN probing accuracy by 1-5%. Additionally, we propose metrics that allow for a comprehensive analysis of the learned features, provide insights into the quality of the representations and help identify collapse patterns.

Related papers

• Reinforcement Learning-Guided Semi-Supervised Learning [20.599506122857328]
  We propose a novel Reinforcement Learning Guided SSL method, RLGSSL, that formulates SSL as a one-armed bandit problem. RLGSSL incorporates a carefully designed reward function that balances the use of labeled and unlabeled data to enhance generalization performance. We demonstrate the effectiveness of RLGSSL through extensive experiments on several benchmark datasets and show that our approach achieves consistent superior performance compared to state-of-the-art SSL methods.
  arXiv  Detail & Related papers  (2024-05-02T21:52:24Z)
• Memorization in Self-Supervised Learning Improves Downstream Generalization [49.42010047574022]
  Self-supervised learning (SSL) has recently received significant attention due to its ability to train high-performance encoders purely on unlabeled data. We propose SSLMem, a framework for defining memorization within SSL.
  arXiv  Detail & Related papers  (2024-01-19T11:32:47Z)
• Understanding Contrastive Learning Through the Lens of Margins [9.443122526245562]
  Self-supervised learning, or SSL, holds the key to expanding the usage of machine learning in real-world tasks. We use margins as a stepping stone for understanding how contrastive learning works at a deeper level.
  arXiv  Detail & Related papers  (2023-06-20T13:28:27Z)
• Augmentation-aware Self-supervised Learning with Conditioned Projector [6.720605329045581]
  Self-supervised learning (SSL) is a powerful technique for learning from unlabeled data. We propose to foster sensitivity to characteristics in the representation space by modifying the projector network. Our approach, coined Conditional Augmentation-aware Self-supervised Learning (CASSLE), is directly applicable to typical joint-embedding SSL methods.
  arXiv  Detail & Related papers  (2023-05-31T12:24:06Z)
• Deciphering the Projection Head: Representation Evaluation Self-supervised Learning [6.375931203397043]
  Self-supervised learning (SSL) aims to learn intrinsic features without labels. Projection head always plays an important role in improving the performance of the downstream task. We propose a Representation Evaluation Design (RED) in SSL models in which a shortcut connection between the representation and the projection vectors is built.
  arXiv  Detail & Related papers  (2023-01-28T13:13:53Z)
• Understanding and Improving the Role of Projection Head in Self-Supervised Learning [77.59320917894043]
  Self-supervised learning (SSL) aims to produce useful feature representations without access to human-labeled data annotations. Current contrastive learning approaches append a parametrized projection head to the end of some backbone network to optimize the InfoNCE objective. This raises a fundamental question: Why is a learnable projection head required if we are to discard it after training?
  arXiv  Detail & Related papers  (2022-12-22T05:42:54Z)
• Self-Supervised PPG Representation Learning Shows High Inter-Subject Variability [3.8036939971290007]
  We propose a Self-Supervised Learning (SSL) method with a pretext task of signal reconstruction to learn an informative generalized PPG representation. Results show that in a very limited label data setting (10 samples per class or less), using SSL is beneficial. SSL may pave the way for the broader use of machine learning models on PPG data in label-scarce regimes.
  arXiv  Detail & Related papers  (2022-12-07T19:02:45Z)
• Improving Self-Supervised Learning by Characterizing Idealized Representations [155.1457170539049]
  We prove necessary and sufficient conditions for any task invariant to given data augmentations. For contrastive learning, our framework prescribes simple but significant improvements to previous methods. For non-contrastive learning, we use our framework to derive a simple and novel objective.
  arXiv  Detail & Related papers  (2022-09-13T18:01:03Z)
• On Higher Adversarial Susceptibility of Contrastive Self-Supervised Learning [104.00264962878956]
  Contrastive self-supervised learning (CSL) has managed to match or surpass the performance of supervised learning in image and video classification. It is still largely unknown if the nature of the representation induced by the two learning paradigms is similar. We identify the uniform distribution of data representation over a unit hypersphere in the CSL representation space as the key contributor to this phenomenon. We devise strategies that are simple, yet effective in improving model robustness with CSL training.
  arXiv  Detail & Related papers  (2022-07-22T03:49:50Z)
• Learning Where to Learn in Cross-View Self-Supervised Learning [54.14989750044489]
  Self-supervised learning (SSL) has made enormous progress and largely narrowed the gap with supervised ones. Current methods simply adopt uniform aggregation of pixels for embedding. We present a new approach, Learning Where to Learn (LEWEL), to adaptively aggregate spatial information of features.
  arXiv  Detail & Related papers  (2022-03-28T17:02:42Z)
• Self-supervised Learning is More Robust to Dataset Imbalance [65.84339596595383]
  We investigate self-supervised learning under dataset imbalance. Off-the-shelf self-supervised representations are already more robust to class imbalance than supervised representations. We devise a re-weighted regularization technique that consistently improves the SSL representation quality on imbalanced datasets.
  arXiv  Detail & Related papers  (2021-10-11T06:29:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.