Integrating Prior Knowledge in Contrastive Learning with Kernel

• URL: http://arxiv.org/abs/2206.01646v2
• Date: Tue, 30 May 2023 11:13:06 GMT
• Title: Integrating Prior Knowledge in Contrastive Learning with Kernel
• Authors: Benoit Dufumier, Carlo Alberto Barbano, Robin Louiset, Edouard Duchesnay, Pietro Gori
• Abstract summary: We use kernel theory to propose a novel loss, called decoupled uniformity, that i) allows the integration of prior knowledge and ii) removes the negative-positive coupling in the original InfoNCE loss. In an unsupervised setting, we empirically demonstrate that CL benefits from generative models to improve its representation both on natural and medical images.
• Score: 4.050766659420731
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Data augmentation is a crucial component in unsupervised contrastive learning (CL). It determines how positive samples are defined and, ultimately, the quality of the learned representation. In this work, we open the door to new perspectives for CL by integrating prior knowledge, given either by generative models -- viewed as prior representations -- or weak attributes in the positive and negative sampling. To this end, we use kernel theory to propose a novel loss, called decoupled uniformity, that i) allows the integration of prior knowledge and ii) removes the negative-positive coupling in the original InfoNCE loss. We draw a connection between contrastive learning and conditional mean embedding theory to derive tight bounds on the downstream classification loss. In an unsupervised setting, we empirically demonstrate that CL benefits from generative models to improve its representation both on natural and medical images. In a weakly supervised scenario, our framework outperforms other unconditional and conditional CL approaches.

Related papers

• Masked Reconstruction Contrastive Learning with Information Bottleneck Principle [9.136962881499734]
  Contrastive learning (CL) has shown great power in self-supervised learning. Current CL models are biased to learn only the ability to discriminate positive and negative pairs. We propose the Masked Reconstruction Contrastive Learning(MRCL) model to improve CL models.
  arXiv  Detail & Related papers  (2022-11-15T15:20:52Z)
• Mitigating Forgetting in Online Continual Learning via Contrasting Semantically Distinct Augmentations [22.289830907729705]
  Online continual learning (OCL) aims to enable model learning from a non-stationary data stream to continuously acquire new knowledge as well as retain the learnt one. Main challenge comes from the "catastrophic forgetting" issue -- the inability to well remember the learnt knowledge while learning the new ones.
  arXiv  Detail & Related papers  (2022-11-10T05:29:43Z)
• Fair Contrastive Learning for Facial Attribute Classification [25.436462696033846]
  We propose a new Fair Supervised Contrastive Loss (FSCL) for fair visual representation learning. In this paper, we for the first time analyze unfairness caused by supervised contrastive learning. Our method is robust to the intensity of data bias and effectively works in incomplete supervised settings.
  arXiv  Detail & Related papers  (2022-03-30T11:16:18Z)
• Chaos is a Ladder: A New Theoretical Understanding of Contrastive Learning via Augmentation Overlap [64.60460828425502]
  We propose a new guarantee on the downstream performance of contrastive learning. Our new theory hinges on the insight that the support of different intra-class samples will become more overlapped under aggressive data augmentations. We propose an unsupervised model selection metric ARC that aligns well with downstream accuracy.
  arXiv  Detail & Related papers  (2022-03-25T05:36:26Z)
• When Does Contrastive Learning Preserve Adversarial Robustness from Pretraining to Finetuning? [99.4914671654374]
  We propose AdvCL, a novel adversarial contrastive pretraining framework. We show that AdvCL is able to enhance cross-task robustness transferability without loss of model accuracy and finetuning efficiency.
  arXiv  Detail & Related papers  (2021-11-01T17:59:43Z)
• Complementary Calibration: Boosting General Continual Learning with Collaborative Distillation and Self-Supervision [47.374412281270594]
  General Continual Learning (GCL) aims at learning from non independent and identically distributed stream data. We reveal that the relation and feature deviations are crucial problems for catastrophic forgetting. We propose a Complementary (CoCa) framework by mining the complementary model's outputs and features.
  arXiv  Detail & Related papers  (2021-09-03T06:35:27Z)
• A Low Rank Promoting Prior for Unsupervised Contrastive Learning [108.91406719395417]
  We construct a novel probabilistic graphical model that effectively incorporates the low rank promoting prior into the framework of contrastive learning. Our hypothesis explicitly requires that all the samples belonging to the same instance class lie on the same subspace with small dimension. Empirical evidences show that the proposed algorithm clearly surpasses the state-of-the-art approaches on multiple benchmarks.
  arXiv  Detail & Related papers  (2021-08-05T15:58:25Z)
• Simpler, Faster, Stronger: Breaking The log-K Curse On Contrastive Learners With FlatNCE [104.37515476361405]
  We reveal mathematically why contrastive learners fail in the small-batch-size regime. We present a novel non-native contrastive objective named FlatNCE, which fixes this issue.
  arXiv  Detail & Related papers  (2021-07-02T15:50:43Z)
• Solving Inefficiency of Self-supervised Representation Learning [87.30876679780532]
  Existing contrastive learning methods suffer from very low learning efficiency. Under-clustering and over-clustering problems are major obstacles to learning efficiency. We propose a novel self-supervised learning framework using a median triplet loss.
  arXiv  Detail & Related papers  (2021-04-18T07:47:10Z)
• Robust Pre-Training by Adversarial Contrastive Learning [120.33706897927391]
  Recent work has shown that, when integrated with adversarial training, self-supervised pre-training can lead to state-of-the-art robustness. We improve robustness-aware self-supervised pre-training by learning representations consistent under both data augmentations and adversarial perturbations.
  arXiv  Detail & Related papers  (2020-10-26T04:44:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.