Rethinking Weak Supervision in Helping Contrastive Learning

• URL: http://arxiv.org/abs/2306.04160v1
• Date: Wed, 7 Jun 2023 05:18:27 GMT
• Title: Rethinking Weak Supervision in Helping Contrastive Learning
• Authors: Jingyi Cui, Weiran Huang, Yifei Wang, Yisen Wang
• Abstract summary: We explore the mechanical differences between semi-supervised and noisy-labeled information in helping contrastive learning. Specifically, we investigate the most intuitive paradigm of jointly training supervised and unsupervised contrastive losses. We prove that semi-supervised labels improve the downstream error bound whereas noisy labels have limited effects under such a paradigm.
• Score: 19.5649824524209
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Contrastive learning has shown outstanding performances in both supervised and unsupervised learning, and has recently been introduced to solve weakly supervised learning problems such as semi-supervised learning and noisy label learning. Despite the empirical evidence showing that semi-supervised labels improve the representations of contrastive learning, it remains unknown if noisy supervised information can be directly used in training instead of after manual denoising. Therefore, to explore the mechanical differences between semi-supervised and noisy-labeled information in helping contrastive learning, we establish a unified theoretical framework of contrastive learning under weak supervision. Specifically, we investigate the most intuitive paradigm of jointly training supervised and unsupervised contrastive losses. By translating the weakly supervised information into a similarity graph under the framework of spectral clustering based on the posterior probability of weak labels, we establish the downstream classification error bound. We prove that semi-supervised labels improve the downstream error bound whereas noisy labels have limited effects under such a paradigm. Our theoretical findings here provide new insights for the community to rethink the role of weak supervision in helping contrastive learning.

Related papers

• Cluster-aware Contrastive Learning for Unsupervised Out-of-distribution Detection [0.0]
  Unsupervised out-of-distribution (OOD) Detection aims to separate the samples falling outside the distribution of training data without label information. We propose Cluster-aware Contrastive Learning (CCL) framework for unsupervised OOD detection, which considers both instance-level and semantic-level information.
  arXiv  Detail & Related papers  (2023-02-06T07:21:03Z)
• Label Propagation with Weak Supervision [47.52032178837098]
  We introduce a novel analysis of the classical label propagation algorithm (LPA) (Zhu & Ghahramani, 2002) We provide an error bound that exploits both the local geometric properties of the underlying graph and the quality of the prior information. We demonstrate the ability of our approach on multiple benchmark weakly supervised classification tasks, showing improvements upon existing semi-supervised and weakly supervised methods.
  arXiv  Detail & Related papers  (2022-10-07T14:53:02Z)
• Resolving label uncertainty with implicit posterior models [71.62113762278963]
  We propose a method for jointly inferring labels across a collection of data samples. By implicitly assuming the existence of a generative model for which a differentiable predictor is the posterior, we derive a training objective that allows learning under weak beliefs.
  arXiv  Detail & Related papers  (2022-02-28T18:09:44Z)
• The Power of Contrast for Feature Learning: A Theoretical Analysis [42.20116348668721]
  We show that contrastive learning outperforms the standard autoencoders and generative adversarial networks. We also illustrate the impact of labeled data in supervised contrastive learning.
  arXiv  Detail & Related papers  (2021-10-06T03:10:28Z)
• Co-learning: Learning from Noisy Labels with Self-supervision [28.266156561454327]
  Self-supervised learning works in the absence of labels and thus eliminates the negative impact of noisy labels. Motivated by co-training with both supervised learning view and self-supervised learning view, we propose a simple yet effective method called Co-learning for learning with noisy labels.
  arXiv  Detail & Related papers  (2021-08-05T06:20:51Z)
• Incremental False Negative Detection for Contrastive Learning [95.68120675114878]
  We introduce a novel incremental false negative detection for self-supervised contrastive learning. During contrastive learning, we discuss two strategies to explicitly remove the detected false negatives. Our proposed method outperforms other self-supervised contrastive learning frameworks on multiple benchmarks within a limited compute.
  arXiv  Detail & Related papers  (2021-06-07T15:29:14Z)
• Disambiguation of weak supervision with exponential convergence rates [88.99819200562784]
  In supervised learning, data are annotated with incomplete yet discriminative information. In this paper, we focus on partial labelling, an instance of weak supervision where, from a given input, we are given a set of potential targets. We propose an empirical disambiguation algorithm to recover full supervision from weak supervision.
  arXiv  Detail & Related papers  (2021-02-04T18:14:32Z)
• Can Semantic Labels Assist Self-Supervised Visual Representation Learning? [194.1681088693248]
  We present a new algorithm named Supervised Contrastive Adjustment in Neighborhood (SCAN) In a series of downstream tasks, SCAN achieves superior performance compared to previous fully-supervised and self-supervised methods. Our study reveals that semantic labels are useful in assisting self-supervised methods, opening a new direction for the community.
  arXiv  Detail & Related papers  (2020-11-17T13:25:00Z)
• A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation [63.042651834453544]
  We show that the unsupervised learning of disentangled representations is impossible without inductive biases on both the models and the data. We observe that while the different methods successfully enforce properties "encouraged" by the corresponding losses, well-disentangled models seemingly cannot be identified without supervision. Our results suggest that future work on disentanglement learning should be explicit about the role of inductive biases and (implicit) supervision.
  arXiv  Detail & Related papers  (2020-10-27T10:17:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.