Sharpness & Shift-Aware Self-Supervised Learning

• URL: http://arxiv.org/abs/2305.10252v1
• Date: Wed, 17 May 2023 14:42:16 GMT
• Title: Sharpness & Shift-Aware Self-Supervised Learning
• Authors: Ngoc N. Tran, Son Duong, Hoang Phan, Tung Pham, Dinh Phung, Trung Le
• Abstract summary: Self-supervised learning aims to extract meaningful features from unlabeled data for further downstream tasks. We develop rigorous theories to realize the factors that implicitly influence the general loss of this classification task. We conduct extensive experiments to verify our theoretical findings and demonstrate that sharpness & shift-aware contrastive learning can remarkably boost the performance.
• Score: 17.978849280772092
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Self-supervised learning aims to extract meaningful features from unlabeled data for further downstream tasks. In this paper, we consider classification as a downstream task in phase 2 and develop rigorous theories to realize the factors that implicitly influence the general loss of this classification task. Our theories signify that sharpness-aware feature extractors benefit the classification task in phase 2 and the existing data shift between the ideal (i.e., the ideal one used in theory development) and practical (i.e., the practical one used in implementation) distributions to generate positive pairs also remarkably affects this classification task. Further harvesting these theoretical findings, we propose to minimize the sharpness of the feature extractor and a new Fourier-based data augmentation technique to relieve the data shift in the distributions generating positive pairs, reaching Sharpness & Shift-Aware Contrastive Learning (SSA-CLR). We conduct extensive experiments to verify our theoretical findings and demonstrate that sharpness & shift-aware contrastive learning can remarkably boost the performance as well as obtaining more robust extracted features compared with the baselines.

Related papers

• Features are fate: a theory of transfer learning in high-dimensional regression [23.840251319669907]
  We show that when the target task is well represented by the feature space of the pre-trained model, transfer learning outperforms training from scratch. For this model, we establish rigorously that when the feature space overlap between the source and target tasks is sufficiently strong, both linear transfer and fine-tuning improve performance.
  arXiv  Detail & Related papers  (2024-10-10T17:58:26Z)
• Unsupervised Transfer Learning via Adversarial Contrastive Training [3.227277661633986]
  We propose a novel unsupervised transfer learning approach using adversarial contrastive training (ACT) Our experimental results demonstrate outstanding classification accuracy with both fine-tuned linear probe and K-NN protocol across various datasets.
  arXiv  Detail & Related papers  (2024-08-16T05:11:52Z)
• Hodge-Aware Contrastive Learning [101.56637264703058]
  Simplicial complexes prove effective in modeling data with multiway dependencies. We develop a contrastive self-supervised learning approach for processing simplicial data.
  arXiv  Detail & Related papers  (2023-09-14T00:40:07Z)
• Regularization Through Simultaneous Learning: A Case Study on Plant Classification [0.0]
  This paper introduces Simultaneous Learning, a regularization approach drawing on principles of Transfer Learning and Multi-task Learning. We leverage auxiliary datasets with the target dataset, the UFOP-HVD, to facilitate simultaneous classification guided by a customized loss function. Remarkably, our approach demonstrates superior performance over models without regularization.
  arXiv  Detail & Related papers  (2023-05-22T19:44:57Z)
• On the Trade-off of Intra-/Inter-class Diversity for Supervised Pre-training [72.8087629914444]
  We study the impact of the trade-off between the intra-class diversity (the number of samples per class) and the inter-class diversity (the number of classes) of a supervised pre-training dataset. With the size of the pre-training dataset fixed, the best downstream performance comes with a balance on the intra-/inter-class diversity.
  arXiv  Detail & Related papers  (2023-05-20T16:23:50Z)
• Deep Semi-supervised Learning with Double-Contrast of Features and Semantics [2.2230089845369094]
  This paper proposes an end-to-end deep semi-supervised learning double contrast of semantic and feature. We leverage information theory to explain the rationality of double contrast of semantics and features.
  arXiv  Detail & Related papers  (2022-11-28T09:08:19Z)
• Explaining, Evaluating and Enhancing Neural Networks' Learned Representations [2.1485350418225244]
  We show how explainability can be an aid, rather than an obstacle, towards better and more efficient representations. We employ such attributions to define two novel scores for evaluating the informativeness and the disentanglement of latent embeddings. We show that adopting our proposed scores as constraints during the training of a representation learning task improves the downstream performance of the model.
  arXiv  Detail & Related papers  (2022-02-18T19:00:01Z)
• Adaptive Hierarchical Similarity Metric Learning with Noisy Labels [138.41576366096137]
  We propose an Adaptive Hierarchical Similarity Metric Learning method. It considers two noise-insensitive information, textiti.e., class-wise divergence and sample-wise consistency. Our method achieves state-of-the-art performance compared with current deep metric learning approaches.
  arXiv  Detail & Related papers  (2021-10-29T02:12:18Z)
• Accurate and Robust Feature Importance Estimation under Distribution Shifts [49.58991359544005]
  PRoFILE is a novel feature importance estimation method. We show significant improvements over state-of-the-art approaches, both in terms of fidelity and robustness.
  arXiv  Detail & Related papers  (2020-09-30T05:29:01Z)
• Adversarial Training Reduces Information and Improves Transferability [81.59364510580738]
  Recent results show that features of adversarially trained networks for classification, in addition to being robust, enable desirable properties such as invertibility. We show that the Adversarial Training can improve linear transferability to new tasks, from which arises a new trade-off between transferability of representations and accuracy on the source task.
  arXiv  Detail & Related papers  (2020-07-22T08:30:16Z)
• Deep Semi-supervised Knowledge Distillation for Overlapping Cervical Cell Instance Segmentation [54.49894381464853]
  We propose to leverage both labeled and unlabeled data for instance segmentation with improved accuracy by knowledge distillation. We propose a novel Mask-guided Mean Teacher framework with Perturbation-sensitive Sample Mining. Experiments show that the proposed method improves the performance significantly compared with the supervised method learned from labeled data only.
  arXiv  Detail & Related papers  (2020-07-21T13:27:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.