Can We Break Free from Strong Data Augmentations in Self-Supervised Learning?

• URL: http://arxiv.org/abs/2404.09752v1
• Date: Mon, 15 Apr 2024 12:53:48 GMT
• Title: Can We Break Free from Strong Data Augmentations in Self-Supervised Learning?
• Authors: Shruthi Gowda, Elahe Arani, Bahram Zonooz,
• Abstract summary: Self-supervised learning (SSL) has emerged as a promising solution for addressing the challenge of limited labeled data in deep neural networks (DNNs) We explore SSL behavior across a spectrum of augmentations, revealing their crucial role in shaping SSL model performance and learning mechanisms. We propose a novel learning approach that integrates prior knowledge, with the aim of curtailing the need for extensive data augmentations.
• Score: 18.83003310612038
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Self-supervised learning (SSL) has emerged as a promising solution for addressing the challenge of limited labeled data in deep neural networks (DNNs), offering scalability potential. However, the impact of design dependencies within the SSL framework remains insufficiently investigated. In this study, we comprehensively explore SSL behavior across a spectrum of augmentations, revealing their crucial role in shaping SSL model performance and learning mechanisms. Leveraging these insights, we propose a novel learning approach that integrates prior knowledge, with the aim of curtailing the need for extensive data augmentations and thereby amplifying the efficacy of learned representations. Notably, our findings underscore that SSL models imbued with prior knowledge exhibit reduced texture bias, diminished reliance on shortcuts and augmentations, and improved robustness against both natural and adversarial corruptions. These findings not only illuminate a new direction in SSL research, but also pave the way for enhancing DNN performance while concurrently alleviating the imperative for intensive data augmentation, thereby enhancing scalability and real-world problem-solving capabilities.

Related papers

• Context-Aware Predictive Coding: A Representation Learning Framework for WiFi Sensing [0.0]
  WiFi sensing is an emerging technology that utilizes wireless signals for various sensing applications. In this paper, we introduce a novel SSL framework called Context-Aware Predictive Coding (CAPC) CAPC effectively learns from unlabelled data and adapts to diverse environments. Our evaluations demonstrate that CAPC not only outperforms other SSL methods and supervised approaches, but also achieves superior generalization capabilities.
  arXiv  Detail & Related papers  (2024-09-16T17:59:49Z)
• A Survey of the Self Supervised Learning Mechanisms for Vision Transformers [5.152455218955949]
  The application of self supervised learning (SSL) in vision tasks has gained significant attention. We develop a comprehensive taxonomy of systematically classifying the SSL techniques. We discuss the motivations behind SSL, review popular pre-training tasks, and highlight the challenges and advancements in this field.
  arXiv  Detail & Related papers  (2024-08-30T07:38:28Z)
• Reinforcement Learning-Guided Semi-Supervised Learning [20.599506122857328]
  We propose a novel Reinforcement Learning Guided SSL method, RLGSSL, that formulates SSL as a one-armed bandit problem. RLGSSL incorporates a carefully designed reward function that balances the use of labeled and unlabeled data to enhance generalization performance. We demonstrate the effectiveness of RLGSSL through extensive experiments on several benchmark datasets and show that our approach achieves consistent superior performance compared to state-of-the-art SSL methods.
  arXiv  Detail & Related papers  (2024-05-02T21:52:24Z)
• LLM-DA: Data Augmentation via Large Language Models for Few-Shot Named Entity Recognition [67.96794382040547]
  $LLM-DA$ is a novel data augmentation technique based on large language models (LLMs) for the few-shot NER task. Our approach involves employing 14 contextual rewriting strategies, designing entity replacements of the same type, and incorporating noise injection to enhance robustness.
  arXiv  Detail & Related papers  (2024-02-22T14:19:56Z)
• Learning Objective-Specific Active Learning Strategies with Attentive Neural Processes [72.75421975804132]
  Learning Active Learning (LAL) suggests to learn the active learning strategy itself, allowing it to adapt to the given setting. We propose a novel LAL method for classification that exploits symmetry and independence properties of the active learning problem. Our approach is based on learning from a myopic oracle, which gives our model the ability to adapt to non-standard objectives.
  arXiv  Detail & Related papers  (2023-09-11T14:16:37Z)
• Self-Supervision for Tackling Unsupervised Anomaly Detection: Pitfalls and Opportunities [50.231837687221685]
  Self-supervised learning (SSL) has transformed machine learning and its many real world applications. Unsupervised anomaly detection (AD) has also capitalized on SSL, by self-generating pseudo-anomalies.
  arXiv  Detail & Related papers  (2023-08-28T07:55:01Z)
• LSFSL: Leveraging Shape Information in Few-shot Learning [11.145085584637746]
  Few-shot learning techniques seek to learn the underlying patterns in data using fewer samples, analogous to how humans learn from limited experience. In this limited-data scenario, the challenges associated with deep neural networks, such as shortcut learning and texture bias behaviors, are further exacerbated. We propose LSFSL, which enforces the model to learn more generalizable features utilizing the implicit prior information present in the data.
  arXiv  Detail & Related papers  (2023-04-13T16:59:22Z)
• Does Decentralized Learning with Non-IID Unlabeled Data Benefit from Self Supervision? [51.00034621304361]
  We study decentralized learning with unlabeled data through the lens of self-supervised learning (SSL) We study the effectiveness of contrastive learning algorithms under decentralized learning settings.
  arXiv  Detail & Related papers  (2022-10-20T01:32:41Z)
• On Higher Adversarial Susceptibility of Contrastive Self-Supervised Learning [104.00264962878956]
  Contrastive self-supervised learning (CSL) has managed to match or surpass the performance of supervised learning in image and video classification. It is still largely unknown if the nature of the representation induced by the two learning paradigms is similar. We identify the uniform distribution of data representation over a unit hypersphere in the CSL representation space as the key contributor to this phenomenon. We devise strategies that are simple, yet effective in improving model robustness with CSL training.
  arXiv  Detail & Related papers  (2022-07-22T03:49:50Z)
• Collaborative Intelligence Orchestration: Inconsistency-Based Fusion of Semi-Supervised Learning and Active Learning [60.26659373318915]
  Active learning (AL) and semi-supervised learning (SSL) are two effective, but often isolated, means to alleviate the data-hungry problem. We propose an innovative Inconsistency-based virtual aDvErial algorithm to further investigate SSL-AL's potential superiority. Two real-world case studies visualize the practical industrial value of applying and deploying the proposed data sampling algorithm.
  arXiv  Detail & Related papers  (2022-06-07T13:28:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.