ScoreCL: Augmentation-Adaptive Contrastive Learning via Score-Matching Function

• URL: http://arxiv.org/abs/2306.04175v3
• Date: Mon, 15 Jul 2024 04:21:08 GMT
• Title: ScoreCL: Augmentation-Adaptive Contrastive Learning via Score-Matching Function
• Authors: Jin-Young Kim, Soonwoo Kwon, Hyojun Go, Yunsung Lee, Seungtaek Choi, Hyun-Gyoon Kim,
• Abstract summary: Self-supervised contrastive learning (CL) has achieved state-of-the-art performance in representation learning. We show the generality of our method, referred to as ScoreCL, by consistently improving various CL methods.
• Score: 14.857965612960475
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Self-supervised contrastive learning (CL) has achieved state-of-the-art performance in representation learning by minimizing the distance between positive pairs while maximizing that of negative ones. Recently, it has been verified that the model learns better representation with diversely augmented positive pairs because they enable the model to be more view-invariant. However, only a few studies on CL have considered the difference between augmented views, and have not gone beyond the hand-crafted findings. In this paper, we first observe that the score-matching function can measure how much data has changed from the original through augmentation. With the observed property, every pair in CL can be weighted adaptively by the difference of score values, resulting in boosting the performance of the existing CL method. We show the generality of our method, referred to as ScoreCL, by consistently improving various CL methods, SimCLR, SimSiam, W-MSE, and VICReg, up to 3%p in k-NN evaluation on CIFAR-10, CIFAR-100, and ImageNet-100. Moreover, we have conducted exhaustive experiments and ablations, including results on diverse downstream tasks, comparison with possible baselines, and improvement when used with other proposed augmentation methods. We hope our exploration will inspire more research in exploiting the score matching for CL.

Related papers

• The Hard Positive Truth about Vision-Language Compositionality [64.8065854134201]
  We investigate whether finetuned vision-language models remain invariant to hard positives. We produce a 1,775,259 image-text training set with both hard negative and hard positive captions. Our work suggests the need for future research to rigorously test and improve CLIP's understanding of semantic relationships between related "positive" concepts.
  arXiv  Detail & Related papers  (2024-09-26T15:36:10Z)
• AMU-Tuning: Effective Logit Bias for CLIP-based Few-shot Learning [50.78033979438031]
  We first introduce a unified formulation to analyze CLIP-based few-shot learning methods from a perspective of logit bias. Based on analysis of key components, this paper proposes a novel AMU-Tuning method to learn effective logit bias for CLIP-based few-shot classification.
  arXiv  Detail & Related papers  (2024-04-13T10:46:11Z)
• Decoupled Contrastive Learning for Long-Tailed Recognition [58.255966442426484]
  Supervised Contrastive Loss (SCL) is popular in visual representation learning. In the scenario of long-tailed recognition, where the number of samples in each class is imbalanced, treating two types of positive samples equally leads to the biased optimization for intra-category distance. We propose a patch-based self distillation to transfer knowledge from head to tail classes to relieve the under-representation of tail classes.
  arXiv  Detail & Related papers  (2024-03-10T09:46:28Z)
• Enhancing Contrastive Learning with Efficient Combinatorial Positive Pairing [2.7961972519572442]
  We propose a general multi-view strategy that can improve learning speed and performance of any contrastive or non-contrastive method. In case of ImageNet-100, ECPP boosted SimCLR outperforms supervised learning.
  arXiv  Detail & Related papers  (2024-01-11T08:18:30Z)
• In-context Learning and Gradient Descent Revisited [3.085927389171139]
  We show that even untrained models achieve comparable ICL-GD similarity scores despite not exhibiting ICL. Next, we explore a major discrepancy in the flow of information throughout the model between ICL and GD, which we term Layer Causality. We propose a simple GD-based optimization procedure that respects layer causality, and show it improves similarity scores significantly.
  arXiv  Detail & Related papers  (2023-11-13T21:42:38Z)
• Self Contrastive Learning for Session-based Recommendation [16.69827431125858]
  Self-Contrastive Learning (SCL) is formulated as an objective function that directly promotes a uniform distribution among item representations. SCL consistently improves the performance of state-of-the-art models with statistical significance.
  arXiv  Detail & Related papers  (2023-06-02T04:43:21Z)
• CLIP Itself is a Strong Fine-tuner: Achieving 85.7% and 88.0% Top-1 Accuracy with ViT-B and ViT-L on ImageNet [139.56863124214905]
  We find that fine-tuning performance of CLIP is substantially underestimated. Specifically, CLIP ViT-Base/16 and CLIP ViT-Large/14 can achieve 85.7%,88.0% finetuning Top-1 accuracy on the ImageNet-1K dataset.
  arXiv  Detail & Related papers  (2022-12-12T18:59:59Z)
• Improving Dense Contrastive Learning with Dense Negative Pairs [15.728417548134047]
  We study how to improve the quality of the representations learned by DenseCL by modifying the training scheme and objective function. Our results show 3.5% and 4% mAP improvement over SimCLR and DenseCL in COCO multi-label classification.
  arXiv  Detail & Related papers  (2022-10-11T00:26:59Z)
• Contrastive Attraction and Contrastive Repulsion for Representation Learning [131.72147978462348]
  Contrastive learning (CL) methods learn data representations in a self-supervision manner, where the encoder contrasts each positive sample over multiple negative samples. Recent CL methods have achieved promising results when pretrained on large-scale datasets, such as ImageNet. We propose a doubly CL strategy that separately compares positive and negative samples within their own groups, and then proceeds with a contrast between positive and negative groups.
  arXiv  Detail & Related papers  (2021-05-08T17:25:08Z)
• Contrastive Clustering [57.71729650297379]
  We propose Contrastive Clustering (CC) which explicitly performs the instance- and cluster-level contrastive learning. In particular, CC achieves an NMI of 0.705 (0.431) on the CIFAR-10 (CIFAR-100) dataset, which is an up to 19% (39%) performance improvement compared with the best baseline.
  arXiv  Detail & Related papers  (2020-09-21T08:54:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.