Making Self-supervised Learning Robust to Spurious Correlation via Learning-speed Aware Sampling

• URL: http://arxiv.org/abs/2311.16361v2
• Date: Wed, 29 Nov 2023 23:19:30 GMT
• Title: Making Self-supervised Learning Robust to Spurious Correlation via Learning-speed Aware Sampling
• Authors: Weicheng Zhu, Sheng Liu, Carlos Fernandez-Granda, Narges Razavian
• Abstract summary: Self-supervised learning (SSL) has emerged as a powerful technique for learning rich representations from unlabeled data. In real-world settings, spurious correlations between some attributes (e.g. race, gender and age) and labels for downstream tasks often exist. We propose a learning-speed aware SSL (LA-SSL) approach, in which we sample each training data with a probability that is inversely related to its learning speed.
• Score: 26.444935219428036
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Self-supervised learning (SSL) has emerged as a powerful technique for learning rich representations from unlabeled data. The data representations are able to capture many underlying attributes of data, and be useful in downstream prediction tasks. In real-world settings, spurious correlations between some attributes (e.g. race, gender and age) and labels for downstream tasks often exist, e.g. cancer is usually more prevalent among elderly patients. In this paper, we investigate SSL in the presence of spurious correlations and show that the SSL training loss can be minimized by capturing only a subset of the conspicuous features relevant to those sensitive attributes, despite the presence of other important predictive features for the downstream tasks. To address this issue, we investigate the learning dynamics of SSL and observe that the learning is slower for samples that conflict with such correlations (e.g. elder patients without cancer). Motivated by these findings, we propose a learning-speed aware SSL (LA-SSL) approach, in which we sample each training data with a probability that is inversely related to its learning speed. We evaluate LA-SSL on three datasets that exhibit spurious correlations between different attributes, demonstrating that it improves the robustness of pretrained representations on downstream classification tasks.

Related papers

• A Closer Look at Benchmarking Self-Supervised Pre-training with Image Classification [51.35500308126506]
  Self-supervised learning (SSL) is a machine learning approach where the data itself provides supervision, eliminating the need for external labels. We study how classification-based evaluation protocols for SSL correlate and how well they predict downstream performance on different dataset types.
  arXiv  Detail & Related papers  (2024-07-16T23:17:36Z)
• Views Can Be Deceiving: Improved SSL Through Feature Space Augmentation [27.609748213840138]
  In this work, we explore the impact of spurious features on Self-Supervised Learning (SSL) for visual representation learning. We show that commonly used augmentations in SSL can cause undesired invariances in the image space. We propose LateTVG to remove spurious information from these representations during pre-training, by regularizing later layers of the encoder via pruning.
  arXiv  Detail & Related papers  (2024-05-28T18:42:13Z)
• Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects [84.6945070729684]
  Self-supervised learning (SSL) has recently achieved impressive performance on various time series tasks. This article reviews current state-of-the-art SSL methods for time series data.
  arXiv  Detail & Related papers  (2023-06-16T18:23:10Z)
• Explaining, Analyzing, and Probing Representations of Self-Supervised Learning Models for Sensor-based Human Activity Recognition [2.2082422928825136]
  Self-supervised learning (SSL) frameworks have been extensively applied to sensor-based Human Activity Recognition (HAR) In this paper, we aim to analyze deep representations of two recent SSL frameworks, namely SimCLR and VICReg.
  arXiv  Detail & Related papers  (2023-04-14T07:53:59Z)
• Self-Supervised PPG Representation Learning Shows High Inter-Subject Variability [3.8036939971290007]
  We propose a Self-Supervised Learning (SSL) method with a pretext task of signal reconstruction to learn an informative generalized PPG representation. Results show that in a very limited label data setting (10 samples per class or less), using SSL is beneficial. SSL may pave the way for the broader use of machine learning models on PPG data in label-scarce regimes.
  arXiv  Detail & Related papers  (2022-12-07T19:02:45Z)
• On Higher Adversarial Susceptibility of Contrastive Self-Supervised Learning [104.00264962878956]
  Contrastive self-supervised learning (CSL) has managed to match or surpass the performance of supervised learning in image and video classification. It is still largely unknown if the nature of the representation induced by the two learning paradigms is similar. We identify the uniform distribution of data representation over a unit hypersphere in the CSL representation space as the key contributor to this phenomenon. We devise strategies that are simple, yet effective in improving model robustness with CSL training.
  arXiv  Detail & Related papers  (2022-07-22T03:49:50Z)
• Collaborative Intelligence Orchestration: Inconsistency-Based Fusion of Semi-Supervised Learning and Active Learning [60.26659373318915]
  Active learning (AL) and semi-supervised learning (SSL) are two effective, but often isolated, means to alleviate the data-hungry problem. We propose an innovative Inconsistency-based virtual aDvErial algorithm to further investigate SSL-AL's potential superiority. Two real-world case studies visualize the practical industrial value of applying and deploying the proposed data sampling algorithm.
  arXiv  Detail & Related papers  (2022-06-07T13:28:43Z)
• Self-supervised Learning is More Robust to Dataset Imbalance [65.84339596595383]
  We investigate self-supervised learning under dataset imbalance. Off-the-shelf self-supervised representations are already more robust to class imbalance than supervised representations. We devise a re-weighted regularization technique that consistently improves the SSL representation quality on imbalanced datasets.
  arXiv  Detail & Related papers  (2021-10-11T06:29:56Z)
• Trash to Treasure: Harvesting OOD Data with Cross-Modal Matching for Open-Set Semi-Supervised Learning [101.28281124670647]
  Open-set semi-supervised learning (open-set SSL) investigates a challenging but practical scenario where out-of-distribution (OOD) samples are contained in the unlabeled data. We propose a novel training mechanism that could effectively exploit the presence of OOD data for enhanced feature learning. Our approach substantially lifts the performance on open-set SSL and outperforms the state-of-the-art by a large margin.
  arXiv  Detail & Related papers  (2021-08-12T09:14:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.