Robust Contrastive Learning With Theory Guarantee

• URL: http://arxiv.org/abs/2311.09671v1
• Date: Thu, 16 Nov 2023 08:39:58 GMT
• Title: Robust Contrastive Learning With Theory Guarantee
• Authors: Ngoc N. Tran, Lam Tran, Hoang Phan, Anh Bui, Tung Pham, Toan Tran, Dinh Phung, Trung Le
• Abstract summary: Contrastive learning (CL) is a self-supervised training paradigm that allows us to extract meaningful features without any label information. Our work develops rigorous theories to dissect and identify which components in the unsupervised loss can help improve the robust supervised loss.
• Score: 25.57187964518637
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Contrastive learning (CL) is a self-supervised training paradigm that allows us to extract meaningful features without any label information. A typical CL framework is divided into two phases, where it first tries to learn the features from unlabelled data, and then uses those features to train a linear classifier with the labeled data. While a fair amount of existing theoretical works have analyzed how the unsupervised loss in the first phase can support the supervised loss in the second phase, none has examined the connection between the unsupervised loss and the robust supervised loss, which can shed light on how to construct an effective unsupervised loss for the first phase of CL. To fill this gap, our work develops rigorous theories to dissect and identify which components in the unsupervised loss can help improve the robust supervised loss and conduct proper experiments to verify our findings.

Related papers

• Dual-Head Knowledge Distillation: Enhancing Logits Utilization with an Auxiliary Head [38.898038672237746]
  We introduce a logit-level loss function as a supplement to the widely used probability-level loss function. We find that the amalgamation of the newly introduced logit-level loss and the previous probability-level loss will lead to performance degeneration. We propose a novel method called dual-head knowledge distillation, which partitions the linear classifier into two classification heads responsible for different losses.
  arXiv  Detail & Related papers  (2024-11-13T12:33:04Z)
• SINDER: Repairing the Singular Defects of DINOv2 [61.98878352956125]
  Vision Transformer models trained on large-scale datasets often exhibit artifacts in the patch token they extract. We propose a novel fine-tuning smooth regularization that rectifies structural deficiencies using only a small dataset.
  arXiv  Detail & Related papers  (2024-07-23T20:34:23Z)
• What Makes CLIP More Robust to Long-Tailed Pre-Training Data? A Controlled Study for Transferable Insights [67.72413262980272]
  Severe data imbalance naturally exists among web-scale vision-language datasets. We find CLIP pre-trained thereupon exhibits notable robustness to the data imbalance compared to supervised learning. The robustness and discriminability of CLIP improve with more descriptive language supervision, larger data scale, and broader open-world concepts.
  arXiv  Detail & Related papers  (2024-05-31T17:57:24Z)
• Semi-Supervised End-To-End Contrastive Learning For Time Series Classification [10.635321868623883]
  Time series classification is a critical task in various domains, such as finance, healthcare, and sensor data analysis. We propose an end-to-end model called SLOTS (Semi-supervised Learning fOr Time clasSification)
  arXiv  Detail & Related papers  (2023-10-13T04:22:21Z)
• Rethinking Weak Supervision in Helping Contrastive Learning [19.5649824524209]
  We explore the mechanical differences between semi-supervised and noisy-labeled information in helping contrastive learning. Specifically, we investigate the most intuitive paradigm of jointly training supervised and unsupervised contrastive losses. We prove that semi-supervised labels improve the downstream error bound whereas noisy labels have limited effects under such a paradigm.
  arXiv  Detail & Related papers  (2023-06-07T05:18:27Z)
• Robustness of Unsupervised Representation Learning without Labels [92.90480374344777]
  We propose a family of unsupervised robustness measures, which are model- and task-agnostic and label-free. We validate our results against a linear probe and show that, for MOCOv2, adversarial training results in 3 times higher certified accuracy.
  arXiv  Detail & Related papers  (2022-10-08T18:03:28Z)
• Semi-Supervised Learning with Mutual Distillation for Monocular Depth Estimation [27.782150368174413]
  We build two separate network branches for each loss and distilling each other through the mutual distillation loss function. We conduct experiments to demonstrate the effectiveness of our framework over the latest methods and provide extensive ablation studies.
  arXiv  Detail & Related papers  (2022-03-18T04:28:58Z)
• Adversarial Dual-Student with Differentiable Spatial Warping for Semi-Supervised Semantic Segmentation [70.2166826794421]
  We propose a differentiable geometric warping to conduct unsupervised data augmentation. We also propose a novel adversarial dual-student framework to improve the Mean-Teacher. Our solution significantly improves the performance and state-of-the-art results are achieved on both datasets.
  arXiv  Detail & Related papers  (2022-03-05T17:36:17Z)
• Leveraged Weighted Loss for Partial Label Learning [64.85763991485652]
  Partial label learning deals with data where each instance is assigned with a set of candidate labels, whereas only one of them is true. Despite many methodology studies on learning from partial labels, there still lacks theoretical understandings of their risk consistent properties. We propose a family of loss functions named textitd weighted (LW) loss, which for the first time introduces the leverage parameter $beta$ to consider the trade-off between losses on partial labels and non-partial ones.
  arXiv  Detail & Related papers  (2021-06-10T13:25:13Z)
• A Commentary on the Unsupervised Learning of Disentangled Representations [63.042651834453544]
  The goal of the unsupervised learning of disentangled representations is to separate the independent explanatory factors of variation in the data without access to supervision. We discuss the theoretical result showing that the unsupervised learning of disentangled representations is fundamentally impossible without inductive biases.
  arXiv  Detail & Related papers  (2020-07-28T13:13:45Z)
• Structured Consistency Loss for semi-supervised semantic segmentation [1.4146420810689415]
  The consistency loss has played a key role in solving problems in recent studies on semi-supervised learning. We propose a structured consistency loss to address this limitation of extant studies. We are the first to present the superiority of state-of-the-art semi-supervised learning in semantic segmentation.
  arXiv  Detail & Related papers  (2020-01-14T07:08:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.