The Bad Batches: Enhancing Self-Supervised Learning in Image Classification Through Representative Batch Curation

• URL: http://arxiv.org/abs/2403.19579v1
• Date: Thu, 28 Mar 2024 17:04:07 GMT
• Title: The Bad Batches: Enhancing Self-Supervised Learning in Image Classification Through Representative Batch Curation
• Authors: Ozgu Goksu, Nicolas Pugeault,
• Abstract summary: The pursuit of learning robust representations without human supervision is a longstanding challenge. This paper attempts to alleviate the influence of false positive and false negative pairs by employing pairwise similarity calculations through the Fr'echet ResNet Distance (FRD) The effectiveness of the proposed method is substantiated by empirical results, where a linear classifier trained on self-supervised contrastive representations achieved an impressive 87.74% top-1 accuracy on STL10 and 99.31% on the Flower102 dataset.
• Score: 1.519321208145928
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The pursuit of learning robust representations without human supervision is a longstanding challenge. The recent advancements in self-supervised contrastive learning approaches have demonstrated high performance across various representation learning challenges. However, current methods depend on the random transformation of training examples, resulting in some cases of unrepresentative positive pairs that can have a large impact on learning. This limitation not only impedes the convergence of the learning process but the robustness of the learnt representation as well as requiring larger batch sizes to improve robustness to such bad batches. This paper attempts to alleviate the influence of false positive and false negative pairs by employing pairwise similarity calculations through the Fr\'echet ResNet Distance (FRD), thereby obtaining robust representations from unlabelled data. The effectiveness of the proposed method is substantiated by empirical results, where a linear classifier trained on self-supervised contrastive representations achieved an impressive 87.74\% top-1 accuracy on STL10 and 99.31\% on the Flower102 dataset. These results emphasize the potential of the proposed approach in pushing the boundaries of the state-of-the-art in self-supervised contrastive learning, particularly for image classification tasks.

Related papers

• LeOCLR: Leveraging Original Images for Contrastive Learning of Visual Representations [4.680881326162484]
  Contrastive instance discrimination methods outperform supervised learning in downstream tasks such as image classification and object detection. A common augmentation technique in contrastive learning is random cropping followed by resizing. We introduce LeOCLR, a framework that employs a novel instance discrimination approach and an adapted loss function.
  arXiv  Detail & Related papers  (2024-03-11T15:33:32Z)
• From Pretext to Purpose: Batch-Adaptive Self-Supervised Learning [32.18543787821028]
  This paper proposes an adaptive technique of batch fusion for self-supervised contrastive learning. It achieves state-of-the-art performance under equitable comparisons. We suggest that the proposed method may contribute to the advancement of data-driven self-supervised learning research.
  arXiv  Detail & Related papers  (2023-11-16T15:47:49Z)
• R\'enyiCL: Contrastive Representation Learning with Skew R\'enyi Divergence [78.15455360335925]
  We present a new robust contrastive learning scheme, coined R'enyiCL, which can effectively manage harder augmentations. Our method is built upon the variational lower bound of R'enyi divergence. We show that R'enyi contrastive learning objectives perform innate hard negative sampling and easy positive sampling simultaneously.
  arXiv  Detail & Related papers  (2022-08-12T13:37:05Z)
• Fair Contrastive Learning for Facial Attribute Classification [25.436462696033846]
  We propose a new Fair Supervised Contrastive Loss (FSCL) for fair visual representation learning. In this paper, we for the first time analyze unfairness caused by supervised contrastive learning. Our method is robust to the intensity of data bias and effectively works in incomplete supervised settings.
  arXiv  Detail & Related papers  (2022-03-30T11:16:18Z)
• Understanding Contrastive Learning Requires Incorporating Inductive Biases [64.56006519908213]
  Recent attempts to theoretically explain the success of contrastive learning on downstream tasks prove guarantees depending on properties of em augmentations and the value of em contrastive loss of representations. We demonstrate that such analyses ignore em inductive biases of the function class and training algorithm, even em provably leading to vacuous guarantees in some settings.
  arXiv  Detail & Related papers  (2022-02-28T18:59:20Z)
• Weakly Supervised Contrastive Learning [68.47096022526927]
  We introduce a weakly supervised contrastive learning framework (WCL) to tackle this issue. WCL achieves 65% and 72% ImageNet Top-1 Accuracy using ResNet50, which is even higher than SimCLRv2 with ResNet101.
  arXiv  Detail & Related papers  (2021-10-10T12:03:52Z)
• Contrastive Learning for Fair Representations [50.95604482330149]
  Trained classification models can unintentionally lead to biased representations and predictions. Existing debiasing methods for classification models, such as adversarial training, are often expensive to train and difficult to optimise. We propose a method for mitigating bias by incorporating contrastive learning, in which instances sharing the same class label are encouraged to have similar representations.
  arXiv  Detail & Related papers  (2021-09-22T10:47:51Z)
• Hybrid Generative-Contrastive Representation Learning [32.84066504783469]
  We show that a transformer-based encoder-decoder architecture trained with both contrastive and generative losses can learn highly discriminative and robust representations without hurting the generative performance.
  arXiv  Detail & Related papers  (2021-06-11T04:23:48Z)
• Solving Inefficiency of Self-supervised Representation Learning [87.30876679780532]
  Existing contrastive learning methods suffer from very low learning efficiency. Under-clustering and over-clustering problems are major obstacles to learning efficiency. We propose a novel self-supervised learning framework using a median triplet loss.
  arXiv  Detail & Related papers  (2021-04-18T07:47:10Z)
• Robust Pre-Training by Adversarial Contrastive Learning [120.33706897927391]
  Recent work has shown that, when integrated with adversarial training, self-supervised pre-training can lead to state-of-the-art robustness. We improve robustness-aware self-supervised pre-training by learning representations consistent under both data augmentations and adversarial perturbations.
  arXiv  Detail & Related papers  (2020-10-26T04:44:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.