Unleashing the Power of Contrastive Self-Supervised Visual Models via
Contrast-Regularized Fine-Tuning
- URL: http://arxiv.org/abs/2102.06605v1
- Date: Fri, 12 Feb 2021 16:31:24 GMT
- Title: Unleashing the Power of Contrastive Self-Supervised Visual Models via
Contrast-Regularized Fine-Tuning
- Authors: Yifan Zhang, Bryan Hooi, Dapeng Hu, Jian Liang, Jiashi Feng
- Abstract summary: We investigate whether applying contrastive learning to fine-tuning would bring further benefits.
We propose Contrast-regularized tuning (Core-tuning), a novel approach for fine-tuning contrastive self-supervised visual models.
- Score: 94.35586521144117
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Contrastive self-supervised learning (CSL) leverages unlabeled data to train
models that provide instance-discriminative visual representations uniformly
scattered in the feature space. In deployment, the common practice is to
directly fine-tune models with the cross-entropy loss, which however may not be
an optimal strategy. Although cross-entropy tends to separate inter-class
features, the resulted models still have limited capability of reducing
intra-class feature scattering that inherits from pre-training, and thus may
suffer unsatisfactory performance on downstream tasks. In this paper, we
investigate whether applying contrastive learning to fine-tuning would bring
further benefits, and analytically find that optimizing the supervised
contrastive loss benefits both class-discriminative representation learning and
model optimization during fine-tuning. Inspired by these findings, we propose
Contrast-regularized tuning (Core-tuning), a novel approach for fine-tuning
contrastive self-supervised visual models. Instead of simply adding the
contrastive loss to the objective of fine-tuning, Core-tuning also generates
hard sample pairs for more effective contrastive learning through a novel
feature mixup strategy, as well as improves the generalizability of the model
by smoothing the decision boundary via mixed samples. Extensive experiments on
image classification and semantic segmentation verify the effectiveness of
Core-tuning.
Related papers
- Enhancing Robustness of Vision-Language Models through Orthogonality Learning and Self-Regularization [77.62516752323207]
We introduce an orthogonal fine-tuning method for efficiently fine-tuning pretrained weights and enabling enhanced robustness and generalization.
A self-regularization strategy is further exploited to maintain the stability in terms of zero-shot generalization of VLMs, dubbed OrthSR.
For the first time, we revisit the CLIP and CoOp with our method to effectively improve the model on few-shot image classficiation scenario.
arXiv Detail & Related papers (2024-07-11T10:35:53Z) - Bayesian Learning-driven Prototypical Contrastive Loss for Class-Incremental Learning [42.14439854721613]
We propose a prototypical network with a Bayesian learning-driven contrastive loss (BLCL) tailored specifically for class-incremental learning scenarios.
Our approach dynamically adapts the balance between the cross-entropy and contrastive loss functions with a Bayesian learning technique.
arXiv Detail & Related papers (2024-05-17T19:49:02Z) - Rethinking Prototypical Contrastive Learning through Alignment,
Uniformity and Correlation [24.794022951873156]
We propose to learn Prototypical representation through Alignment, Uniformity and Correlation (PAUC)
Specifically, the ordinary ProtoNCE loss is revised with: (1) an alignment loss that pulls embeddings from positive prototypes together; (2) a loss that distributes the prototypical level features uniformly; (3) a correlation loss that increases the diversity and discriminability between prototypical level features.
arXiv Detail & Related papers (2022-10-18T22:33:12Z) - PointACL:Adversarial Contrastive Learning for Robust Point Clouds
Representation under Adversarial Attack [73.3371797787823]
Adversarial contrastive learning (ACL) is considered an effective way to improve the robustness of pre-trained models.
We present our robust aware loss function to train self-supervised contrastive learning framework adversarially.
We validate our method, PointACL on downstream tasks, including 3D classification and 3D segmentation with multiple datasets.
arXiv Detail & Related papers (2022-09-14T22:58:31Z) - Revisiting Consistency Regularization for Semi-Supervised Learning [80.28461584135967]
We propose an improved consistency regularization framework by a simple yet effective technique, FeatDistLoss.
Experimental results show that our model defines a new state of the art for various datasets and settings.
arXiv Detail & Related papers (2021-12-10T20:46:13Z) - Contrastive Learning for Fair Representations [50.95604482330149]
Trained classification models can unintentionally lead to biased representations and predictions.
Existing debiasing methods for classification models, such as adversarial training, are often expensive to train and difficult to optimise.
We propose a method for mitigating bias by incorporating contrastive learning, in which instances sharing the same class label are encouraged to have similar representations.
arXiv Detail & Related papers (2021-09-22T10:47:51Z) - Orthogonal Ensemble Networks for Biomedical Image Segmentation [10.011414604407681]
We introduce Orthogonal Ensemble Networks (OEN), a novel framework to explicitly enforce model diversity.
We benchmark the proposed framework in two challenging brain lesion segmentation tasks.
The experimental results show that our approach produces more robust and well-calibrated ensemble models.
arXiv Detail & Related papers (2021-05-22T23:44:55Z) - Solving Inefficiency of Self-supervised Representation Learning [87.30876679780532]
Existing contrastive learning methods suffer from very low learning efficiency.
Under-clustering and over-clustering problems are major obstacles to learning efficiency.
We propose a novel self-supervised learning framework using a median triplet loss.
arXiv Detail & Related papers (2021-04-18T07:47:10Z) - On the Benefits of Models with Perceptually-Aligned Gradients [8.427953227125148]
We show that interpretable and perceptually aligned gradients are present even in models that do not show high robustness to adversarial attacks.
We leverage models with interpretable perceptually-aligned features and show that adversarial training with low max-perturbation bound can improve the performance of models for zero-shot and weakly supervised localization tasks.
arXiv Detail & Related papers (2020-05-04T14:05:38Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.