Learning by Sorting: Self-supervised Learning with Group Ordering Constraints

• URL: http://arxiv.org/abs/2301.02009v2
• Date: Sat, 19 Aug 2023 01:07:11 GMT
• Title: Learning by Sorting: Self-supervised Learning with Group Ordering Constraints
• Authors: Nina Shvetsova, Felix Petersen, Anna Kukleva, Bernt Schiele, Hilde Kuehne
• Abstract summary: This paper proposes a new variation of the contrastive learning objective, Group Ordering Constraints (GroCo) It exploits the idea of sorting the distances of positive and negative pairs and computing the respective loss based on how many positive pairs have a larger distance than the negative pairs, and thus are not ordered correctly. We evaluate the proposed formulation on various self-supervised learning benchmarks and show that it not only leads to improved results compared to vanilla contrastive learning but also shows competitive performance to comparable methods in linear probing and outperforms current methods in k-NN performance.
• Score: 75.89238437237445
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Contrastive learning has become an important tool in learning representations from unlabeled data mainly relying on the idea of minimizing distance between positive data pairs, e.g., views from the same images, and maximizing distance between negative data pairs, e.g., views from different images. This paper proposes a new variation of the contrastive learning objective, Group Ordering Constraints (GroCo), that leverages the idea of sorting the distances of positive and negative pairs and computing the respective loss based on how many positive pairs have a larger distance than the negative pairs, and thus are not ordered correctly. To this end, the GroCo loss is based on differentiable sorting networks, which enable training with sorting supervision by matching a differentiable permutation matrix, which is produced by sorting a given set of scores, to a respective ground truth permutation matrix. Applying this idea to groupwise pre-ordered inputs of multiple positive and negative pairs allows introducing the GroCo loss with implicit emphasis on strong positives and negatives, leading to better optimization of the local neighborhood. We evaluate the proposed formulation on various self-supervised learning benchmarks and show that it not only leads to improved results compared to vanilla contrastive learning but also shows competitive performance to comparable methods in linear probing and outperforms current methods in k-NN performance.

Related papers

• Smoothed Graph Contrastive Learning via Seamless Proximity Integration [30.247207861739245]
  Graph contrastive learning (GCL) aligns node representations by classifying node pairs into positives and negatives. We present a Smoothed Graph Contrastive Learning model (SGCL) that injects proximity information associated with positive/negative pairs in the contrastive loss. The proposed SGCL adjusts the penalties associated with node pairs in contrastive loss by incorporating three distinct smoothing techniques.
  arXiv  Detail & Related papers  (2024-02-23T11:32:46Z)
• Bipartite Ranking Fairness through a Model Agnostic Ordering Adjustment [54.179859639868646]
  We propose a model agnostic post-processing framework xOrder for achieving fairness in bipartite ranking. xOrder is compatible with various classification models and ranking fairness metrics, including supervised and unsupervised fairness metrics. We evaluate our proposed algorithm on four benchmark data sets and two real-world patient electronic health record repositories.
  arXiv  Detail & Related papers  (2023-07-27T07:42:44Z)
• Deep Negative Correlation Classification [82.45045814842595]
  Existing deep ensemble methods naively train many different models and then aggregate their predictions. We propose deep negative correlation classification (DNCC) DNCC yields a deep classification ensemble where the individual estimator is both accurate and negatively correlated.
  arXiv  Detail & Related papers  (2022-12-14T07:35:20Z)
• Twin Contrastive Learning for Online Clustering [15.9794051341163]
  This paper proposes to perform online clustering by conducting twin contrastive learning (TCL) at the instance and cluster level. We find that when the data is projected into a feature space with a dimensionality of the target cluster number, the rows and columns of its feature matrix correspond to the instance and cluster representation.
  arXiv  Detail & Related papers  (2022-10-21T02:12:48Z)
• Contrasting quadratic assignments for set-based representation learning [5.142415132534397]
  standard approach to contrastive learning is to maximize the agreement between different views of the data. In this work, we note that the approach of considering individual pairs cannot account for both intra-set and inter-set similarities. We propose to go beyond contrasting individual pairs of objects by focusing on contrasting objects as sets.
  arXiv  Detail & Related papers  (2022-05-31T14:14:36Z)
• Learning to Hash Naturally Sorts [84.90210592082829]
  We introduce Naturally-Sorted Hashing (NSH) to train a deep hashing model with sorted results end-to-end. NSH sort the Hamming distances of samples' hash codes and accordingly gather their latent representations for self-supervised training. We describe a novel Sorted Noise-Contrastive Estimation (SortedNCE) loss that selectively picks positive and negative samples for contrastive learning.
  arXiv  Detail & Related papers  (2022-01-31T16:19:02Z)
• Not All Negatives are Equal: Label-Aware Contrastive Loss for Fine-grained Text Classification [0.0]
  We analyse the contrastive fine-tuning of pre-trained language models on two fine-grained text classification tasks. We adaptively embed class relationships into a contrastive objective function to help differently weigh the positives and negatives. We find that Label-aware Contrastive Loss outperforms previous contrastive methods.
  arXiv  Detail & Related papers  (2021-09-12T04:19:17Z)
• Contrastive Attraction and Contrastive Repulsion for Representation Learning [131.72147978462348]
  Contrastive learning (CL) methods learn data representations in a self-supervision manner, where the encoder contrasts each positive sample over multiple negative samples. Recent CL methods have achieved promising results when pretrained on large-scale datasets, such as ImageNet. We propose a doubly CL strategy that separately compares positive and negative samples within their own groups, and then proceeds with a contrast between positive and negative groups.
  arXiv  Detail & Related papers  (2021-05-08T17:25:08Z)
• Contrastive Clustering [57.71729650297379]
  We propose Contrastive Clustering (CC) which explicitly performs the instance- and cluster-level contrastive learning. In particular, CC achieves an NMI of 0.705 (0.431) on the CIFAR-10 (CIFAR-100) dataset, which is an up to 19% (39%) performance improvement compared with the best baseline.
  arXiv  Detail & Related papers  (2020-09-21T08:54:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.