A Framework For Contrastive Self-Supervised Learning And Designing A New Approach

• URL: http://arxiv.org/abs/2009.00104v1
• Date: Mon, 31 Aug 2020 21:11:48 GMT
• Title: A Framework For Contrastive Self-Supervised Learning And Designing A New Approach
• Authors: William Falcon, Kyunghyun Cho
• Abstract summary: Contrastive self-supervised learning (CSL) is an approach to learn useful representations by solving a pretext task. We present a conceptual framework that characterizes CSL approaches in five aspects.
• Score: 78.62764948912502
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Contrastive self-supervised learning (CSL) is an approach to learn useful representations by solving a pretext task that selects and compares anchor, negative and positive (APN) features from an unlabeled dataset. We present a conceptual framework that characterizes CSL approaches in five aspects (1) data augmentation pipeline, (2) encoder selection, (3) representation extraction, (4) similarity measure, and (5) loss function. We analyze three leading CSL approaches--AMDIM, CPC, and SimCLR--, and show that despite different motivations, they are special cases under this framework. We show the utility of our framework by designing Yet Another DIM (YADIM) which achieves competitive results on CIFAR-10, STL-10 and ImageNet, and is more robust to the choice of encoder and the representation extraction strategy. To support ongoing CSL research, we release the PyTorch implementation of this conceptual framework along with standardized implementations of AMDIM, CPC (V2), SimCLR, BYOL, Moco (V2) and YADIM.

Related papers

• In-Context Learning for Few-Shot Nested Named Entity Recognition [53.55310639969833]
  We introduce an effective and innovative ICL framework for the setting of few-shot nested NER. We improve the ICL prompt by devising a novel example demonstration selection mechanism, EnDe retriever. In EnDe retriever, we employ contrastive learning to perform three types of representation learning, in terms of semantic similarity, boundary similarity, and label similarity.
  arXiv  Detail & Related papers  (2024-02-02T06:57:53Z)
• CSL: Class-Agnostic Structure-Constrained Learning for Segmentation Including the Unseen [62.72636247006293]
  Class-Agnostic Structure-Constrained Learning is a plug-in framework that can integrate with existing methods. We propose soft assignment and mask split methodologies that enhance OOD object segmentation. Empirical evaluations demonstrate CSL's prowess in boosting the performance of existing algorithms spanning OOD segmentation, ZS3, and DA segmentation.
  arXiv  Detail & Related papers  (2023-12-09T11:06:18Z)
• Semantic Positive Pairs for Enhancing Visual Representation Learning of Instance Discrimination methods [4.680881326162484]
  Self-supervised learning algorithms (SSL) based on instance discrimination have shown promising results. We propose an approach to identify those images with similar semantic content and treat them as positive instances. We run experiments on three benchmark datasets: ImageNet, STL-10 and CIFAR-10 with different instance discrimination SSL approaches.
  arXiv  Detail & Related papers  (2023-06-28T11:47:08Z)
• Deciphering the Projection Head: Representation Evaluation Self-supervised Learning [6.375931203397043]
  Self-supervised learning (SSL) aims to learn intrinsic features without labels. Projection head always plays an important role in improving the performance of the downstream task. We propose a Representation Evaluation Design (RED) in SSL models in which a shortcut connection between the representation and the projection vectors is built.
  arXiv  Detail & Related papers  (2023-01-28T13:13:53Z)
• Decoupled Adversarial Contrastive Learning for Self-supervised Adversarial Robustness [69.39073806630583]
  Adversarial training (AT) for robust representation learning and self-supervised learning (SSL) for unsupervised representation learning are two active research fields. We propose a two-stage framework termed Decoupled Adversarial Contrastive Learning (DeACL)
  arXiv  Detail & Related papers  (2022-07-22T06:30:44Z)
• 3D Human Action Representation Learning via Cross-View Consistency Pursuit [52.19199260960558]
  We propose a Cross-view Contrastive Learning framework for unsupervised 3D skeleton-based action Representation (CrosSCLR) CrosSCLR consists of both single-view contrastive learning (SkeletonCLR) and cross-view consistent knowledge mining (CVC-KM) modules, integrated in a collaborative learning manner.
  arXiv  Detail & Related papers  (2021-04-29T16:29:41Z)
• Spatial-Temporal Multi-Cue Network for Continuous Sign Language Recognition [141.24314054768922]
  We propose a spatial-temporal multi-cue (STMC) network to solve the vision-based sequence learning problem. To validate the effectiveness, we perform experiments on three large-scale CSLR benchmarks.
  arXiv  Detail & Related papers  (2020-02-08T15:38:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.