Active Self-Supervised Learning: A Few Low-Cost Relationships Are All You Need

• URL: http://arxiv.org/abs/2303.15256v2
• Date: Fri, 29 Sep 2023 08:30:32 GMT
• Title: Active Self-Supervised Learning: A Few Low-Cost Relationships Are All You Need
• Authors: Vivien Cabannes, Leon Bottou, Yann Lecun, Randall Balestriero
• Abstract summary: Self-Supervised Learning (SSL) has emerged as the solution of choice to learn transferable representations from unlabeled data. In this work, we formalize and generalize this principle through Positive Active Learning (PAL) where an oracle queries semantic relationships between samples. First, it unveils a theoretically grounded learning framework beyond SSL, based on similarity graphs, that can be extended to tackle supervised and semi-supervised learning depending on the employed oracle. Second, it provides a consistent algorithm to embed a priori knowledge, e.g. some observed labels, into any SSL losses without any change in the training pipeline.
• Score: 34.013568381942775
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Self-Supervised Learning (SSL) has emerged as the solution of choice to learn transferable representations from unlabeled data. However, SSL requires to build samples that are known to be semantically akin, i.e. positive views. Requiring such knowledge is the main limitation of SSL and is often tackled by ad-hoc strategies e.g. applying known data-augmentations to the same input. In this work, we formalize and generalize this principle through Positive Active Learning (PAL) where an oracle queries semantic relationships between samples. PAL achieves three main objectives. First, it unveils a theoretically grounded learning framework beyond SSL, based on similarity graphs, that can be extended to tackle supervised and semi-supervised learning depending on the employed oracle. Second, it provides a consistent algorithm to embed a priori knowledge, e.g. some observed labels, into any SSL losses without any change in the training pipeline. Third, it provides a proper active learning framework yielding low-cost solutions to annotate datasets, arguably bringing the gap between theory and practice of active learning that is based on simple-to-answer-by-non-experts queries of semantic relationships between inputs.

Related papers

• FlexSSL : A Generic and Efficient Framework for Semi-Supervised Learning [19.774959310191623]
  We develop a generic and efficient learning framework called FlexSSL. We show that FlexSSL can consistently enhance the performance of semi-supervised learning algorithms.
  arXiv  Detail & Related papers  (2023-12-28T08:31:56Z)
• A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trends [82.64268080902742]
  Self-supervised learning (SSL) aims to learn discriminative features from unlabeled data without relying on human-annotated labels. SSL has garnered significant attention recently, leading to the development of numerous related algorithms. This paper presents a review of diverse SSL methods, encompassing algorithmic aspects, application domains, three key trends, and open research questions.
  arXiv  Detail & Related papers  (2023-01-13T14:41:05Z)
• FUSSL: Fuzzy Uncertain Self Supervised Learning [8.31483061185317]
  Self supervised learning (SSL) has become a very successful technique to harness the power of unlabeled data, with no annotation effort. In this paper, for the first time, we recognize the fundamental limits of SSL coming from the use of a single-supervisory signal. We propose a robust and general standard hierarchical learning/training protocol for any SSL baseline.
  arXiv  Detail & Related papers  (2022-10-28T01:06:10Z)
• Decoupled Adversarial Contrastive Learning for Self-supervised Adversarial Robustness [69.39073806630583]
  Adversarial training (AT) for robust representation learning and self-supervised learning (SSL) for unsupervised representation learning are two active research fields. We propose a two-stage framework termed Decoupled Adversarial Contrastive Learning (DeACL)
  arXiv  Detail & Related papers  (2022-07-22T06:30:44Z)
• Knowledge-driven Active Learning [70.37119719069499]
  Active learning strategies aim at minimizing the amount of labelled data required to train a Deep Learning model. Most active strategies are based on uncertain sample selection, and even often restricted to samples lying close to the decision boundary. Here we propose to take into consideration common domain-knowledge and enable non-expert users to train a model with fewer samples.
  arXiv  Detail & Related papers  (2021-10-15T06:11:53Z)
• Self-supervised Learning is More Robust to Dataset Imbalance [65.84339596595383]
  We investigate self-supervised learning under dataset imbalance. Off-the-shelf self-supervised representations are already more robust to class imbalance than supervised representations. We devise a re-weighted regularization technique that consistently improves the SSL representation quality on imbalanced datasets.
  arXiv  Detail & Related papers  (2021-10-11T06:29:56Z)
• The Role of Global Labels in Few-Shot Classification and How to Infer Them [55.64429518100676]
  Few-shot learning is a central problem in meta-learning, where learners must quickly adapt to new tasks. We propose Meta Label Learning (MeLa), a novel algorithm that infers global labels and obtains robust few-shot models via standard classification.
  arXiv  Detail & Related papers  (2021-08-09T14:07:46Z)
• ReSSL: Relational Self-Supervised Learning with Weak Augmentation [68.47096022526927]
  Self-supervised learning has achieved great success in learning visual representations without data annotations. We introduce a novel relational SSL paradigm that learns representations by modeling the relationship between different instances. Our proposed ReSSL significantly outperforms the previous state-of-the-art algorithms in terms of both performance and training efficiency.
  arXiv  Detail & Related papers  (2021-07-20T06:53:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.