Demystifying How Self-Supervised Features Improve Training from Noisy Labels

• URL: http://arxiv.org/abs/2110.09022v1
• Date: Mon, 18 Oct 2021 05:41:57 GMT
• Title: Demystifying How Self-Supervised Features Improve Training from Noisy Labels
• Authors: Hao Cheng, Zhaowei Zhu, Xing Sun, Yang Liu
• Abstract summary: We study why and how self-supervised features help networks resist label noise. Our result shows that, given a quality encoder pre-trained from SSL, a simple linear layer trained by the cross-entropy loss is theoretically robust to symmetric label noise.
• Score: 16.281091780103736
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The advancement of self-supervised learning (SSL) motivates researchers to apply SSL on other tasks such as learning with noisy labels. Recent literature indicates that methods built on SSL features can substantially improve the performance of learning with noisy labels. Nonetheless, the deeper reasons why (and how) SSL features benefit the training from noisy labels are less understood. In this paper, we study why and how self-supervised features help networks resist label noise using both theoretical analyses and numerical experiments. Our result shows that, given a quality encoder pre-trained from SSL, a simple linear layer trained by the cross-entropy loss is theoretically robust to symmetric label noise. Further, we provide insights for how knowledge distilled from SSL features can alleviate the over-fitting problem. We hope our work provides a better understanding for learning with noisy labels from the perspective of self-supervised learning and can potentially serve as a guideline for further research. Code is available at github.com/UCSC-REAL/SelfSup_NoisyLabel.

Related papers

• Reinforcement Learning-Guided Semi-Supervised Learning [20.599506122857328]
  We propose a novel Reinforcement Learning Guided SSL method, RLGSSL, that formulates SSL as a one-armed bandit problem. RLGSSL incorporates a carefully designed reward function that balances the use of labeled and unlabeled data to enhance generalization performance. We demonstrate the effectiveness of RLGSSL through extensive experiments on several benchmark datasets and show that our approach achieves consistent superior performance compared to state-of-the-art SSL methods.
  arXiv  Detail & Related papers  (2024-05-02T21:52:24Z)
• Memorization in Self-Supervised Learning Improves Downstream Generalization [49.42010047574022]
  Self-supervised learning (SSL) has recently received significant attention due to its ability to train high-performance encoders purely on unlabeled data. We propose SSLMem, a framework for defining memorization within SSL.
  arXiv  Detail & Related papers  (2024-01-19T11:32:47Z)
• Making Self-supervised Learning Robust to Spurious Correlation via Learning-speed Aware Sampling [26.444935219428036]
  Self-supervised learning (SSL) has emerged as a powerful technique for learning rich representations from unlabeled data. In real-world settings, spurious correlations between some attributes (e.g. race, gender and age) and labels for downstream tasks often exist. We propose a learning-speed aware SSL (LA-SSL) approach, in which we sample each training data with a probability that is inversely related to its learning speed.
  arXiv  Detail & Related papers  (2023-11-27T22:52:45Z)
• A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trends [82.64268080902742]
  Self-supervised learning (SSL) aims to learn discriminative features from unlabeled data without relying on human-annotated labels. SSL has garnered significant attention recently, leading to the development of numerous related algorithms. This paper presents a review of diverse SSL methods, encompassing algorithmic aspects, application domains, three key trends, and open research questions.
  arXiv  Detail & Related papers  (2023-01-13T14:41:05Z)
• Robust Deep Semi-Supervised Learning: A Brief Introduction [63.09703308309176]
  Semi-supervised learning (SSL) aims to improve learning performance by leveraging unlabeled data when labels are insufficient. SSL with deep models has proven to be successful on standard benchmark tasks. However, they are still vulnerable to various robustness threats in real-world applications.
  arXiv  Detail & Related papers  (2022-02-12T04:16:41Z)
• Augmented Contrastive Self-Supervised Learning for Audio Invariant Representations [28.511060004984895]
  We propose an augmented contrastive SSL framework to learn invariant representations from unlabeled data. Our method applies various perturbations to the unlabeled input data and utilizes contrastive learning to learn representations robust to such perturbations.
  arXiv  Detail & Related papers  (2021-12-21T02:50:53Z)
• Constrained Mean Shift for Representation Learning [17.652439157554877]
  We develop a non-contrastive representation learning method that can exploit additional knowledge. Our main idea is to generalize the mean-shift algorithm by constraining the search space of nearest neighbors. We show that it is possible to use the noisy constraint across modalities to train self-supervised video models.
  arXiv  Detail & Related papers  (2021-10-19T23:14:23Z)
• Self-supervised Learning is More Robust to Dataset Imbalance [65.84339596595383]
  We investigate self-supervised learning under dataset imbalance. Off-the-shelf self-supervised representations are already more robust to class imbalance than supervised representations. We devise a re-weighted regularization technique that consistently improves the SSL representation quality on imbalanced datasets.
  arXiv  Detail & Related papers  (2021-10-11T06:29:56Z)
• Understanding (Generalized) Label Smoothing when Learning with Noisy Labels [57.37057235894054]
  Label smoothing (LS) is an arising learning paradigm that uses the positively weighted average of both the hard training labels and uniformly distributed soft labels. We provide understandings for the properties of generalized label smoothing (GLS) when learning with noisy labels.
  arXiv  Detail & Related papers  (2021-06-08T07:32:29Z)
• Noisy Labels Can Induce Good Representations [53.47668632785373]
  We study how architecture affects learning with noisy labels. We show that training with noisy labels can induce useful hidden representations, even when the model generalizes poorly. This finding leads to a simple method to improve models trained on noisy labels.
  arXiv  Detail & Related papers  (2020-12-23T18:58:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.