CARTL: Cooperative Adversarially-Robust Transfer Learning
- URL: http://arxiv.org/abs/2106.06667v1
- Date: Sat, 12 Jun 2021 02:29:55 GMT
- Title: CARTL: Cooperative Adversarially-Robust Transfer Learning
- Authors: Dian Chen, Hongxin Hu, Qian Wang, Yinli Li, Cong Wang, Chao Shen, Qi
Li
- Abstract summary: In deep learning, a typical strategy for transfer learning is to freeze the early layers of a pre-trained model and fine-tune the rest of its layers on the target domain.
We propose a novel cooperative adversarially-robust transfer learning (CARTL) by pre-training the model via feature distance minimization and fine-tuning the pre-trained model with non-expansive fine-tuning for target domain tasks.
- Score: 22.943270371841226
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Transfer learning eases the burden of training a well-performed model from
scratch, especially when training data is scarce and computation power is
limited. In deep learning, a typical strategy for transfer learning is to
freeze the early layers of a pre-trained model and fine-tune the rest of its
layers on the target domain. Previous work focuses on the accuracy of the
transferred model but neglects the transfer of adversarial robustness. In this
work, we first show that transfer learning improves the accuracy on the target
domain but degrades the inherited robustness of the target model. To address
such a problem, we propose a novel cooperative adversarially-robust transfer
learning (CARTL) by pre-training the model via feature distance minimization
and fine-tuning the pre-trained model with non-expansive fine-tuning for target
domain tasks. Empirical results show that CARTL improves the inherited
robustness by about 28% at most compared with the baseline with the same degree
of accuracy. Furthermore, we study the relationship between the batch
normalization (BN) layers and the robustness in the context of transfer
learning, and we reveal that freezing BN layers can further boost the
robustness transfer.
Related papers
- Robust Transfer Learning with Unreliable Source Data [13.276850367115333]
We introduce a novel quantity called the ''ambiguity level'' that measures the discrepancy between the target and source regression functions.
We propose a simple transfer learning procedure, and establish a general theorem that shows how this new quantity is related to the transferability of learning.
arXiv Detail & Related papers (2023-10-06T21:50:21Z) - Analysis of Task Transferability in Large Pre-trained Classifiers [11.517862889784293]
We analyze the transfer of performance for classification tasks, when only the last linear layer of the source model is fine-tuned on the target task.
We propose a novel Task Transfer Analysis approach that transforms the source distribution (and classifier) by changing the class prior distribution, label, and feature spaces.
We perform a large-scale empirical study by using state-of-the-art pre-trained models and demonstrate the effectiveness of our bound and optimization at predicting transferability.
arXiv Detail & Related papers (2023-07-03T08:06:22Z) - Optimal transfer protocol by incremental layer defrosting [66.76153955485584]
Transfer learning is a powerful tool enabling model training with limited amounts of data.
The simplest transfer learning protocol is based on freezing" the feature-extractor layers of a network pre-trained on a data-rich source task.
We show that this protocol is often sub-optimal and the largest performance gain may be achieved when smaller portions of the pre-trained network are kept frozen.
arXiv Detail & Related papers (2023-03-02T17:32:11Z) - Frustratingly Easy Transferability Estimation [64.42879325144439]
We propose a simple, efficient, and effective transferability measure named TransRate.
TransRate measures the transferability as the mutual information between the features of target examples extracted by a pre-trained model and labels of them.
Despite its extraordinary simplicity in 10 lines of codes, TransRate performs remarkably well in extensive evaluations on 22 pre-trained models and 16 downstream tasks.
arXiv Detail & Related papers (2021-06-17T10:27:52Z) - Towards Accurate Knowledge Transfer via Target-awareness Representation
Disentanglement [56.40587594647692]
We propose a novel transfer learning algorithm, introducing the idea of Target-awareness REpresentation Disentanglement (TRED)
TRED disentangles the relevant knowledge with respect to the target task from the original source model and used as a regularizer during fine-tuning the target model.
Experiments on various real world datasets show that our method stably improves the standard fine-tuning by more than 2% in average.
arXiv Detail & Related papers (2020-10-16T17:45:08Z) - Unsupervised Transfer Learning for Spatiotemporal Predictive Networks [90.67309545798224]
We study how to transfer knowledge from a zoo of unsupervisedly learned models towards another network.
Our motivation is that models are expected to understand complex dynamics from different sources.
Our approach yields significant improvements on three benchmarks fortemporal prediction, and benefits the target even from less relevant ones.
arXiv Detail & Related papers (2020-09-24T15:40:55Z) - Do Adversarially Robust ImageNet Models Transfer Better? [102.09335596483695]
adversarially robust models often perform better than their standard-trained counterparts when used for transfer learning.
Our results are consistent with (and in fact, add to) recent hypotheses stating that robustness leads to improved feature representations.
arXiv Detail & Related papers (2020-07-16T17:42:40Z) - Adversarially-Trained Deep Nets Transfer Better: Illustration on Image
Classification [53.735029033681435]
Transfer learning is a powerful methodology for adapting pre-trained deep neural networks on image recognition tasks to new domains.
In this work, we demonstrate that adversarially-trained models transfer better than non-adversarially-trained models.
arXiv Detail & Related papers (2020-07-11T22:48:42Z) - Minimax Lower Bounds for Transfer Learning with Linear and One-hidden
Layer Neural Networks [27.44348371795822]
We develop a statistical minimax framework to characterize the limits of transfer learning.
We derive a lower-bound for the target generalization error achievable by any algorithm as a function of the number of labeled source and target data.
arXiv Detail & Related papers (2020-06-16T22:49:26Z) - Renofeation: A Simple Transfer Learning Method for Improved Adversarial
Robustness [26.73248223512572]
A recent adversarial attack can successfully deceive models trained with transfer learning via end-to-end fine-tuning.
This raises security concerns for many industrial applications.
We propose noisy feature distillation, a new transfer learning method.
arXiv Detail & Related papers (2020-02-07T20:07:22Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.