Do Adversarially Robust ImageNet Models Transfer Better?

• URL: http://arxiv.org/abs/2007.08489v2
• Date: Tue, 8 Dec 2020 01:56:10 GMT
• Title: Do Adversarially Robust ImageNet Models Transfer Better?
• Authors: Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, Aleksander Madry
• Abstract summary: adversarially robust models often perform better than their standard-trained counterparts when used for transfer learning. Our results are consistent with (and in fact, add to) recent hypotheses stating that robustness leads to improved feature representations.
• Score: 102.09335596483695
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Transfer learning is a widely-used paradigm in deep learning, where models pre-trained on standard datasets can be efficiently adapted to downstream tasks. Typically, better pre-trained models yield better transfer results, suggesting that initial accuracy is a key aspect of transfer learning performance. In this work, we identify another such aspect: we find that adversarially robust models, while less accurate, often perform better than their standard-trained counterparts when used for transfer learning. Specifically, we focus on adversarially robust ImageNet classifiers, and show that they yield improved accuracy on a standard suite of downstream classification tasks. Further analysis uncovers more differences between robust and standard models in the context of transfer learning. Our results are consistent with (and in fact, add to) recent hypotheses stating that robustness leads to improved feature representations. Our code and models are available at https://github.com/Microsoft/robust-models-transfer .

Related papers

• Fantastic Gains and Where to Find Them: On the Existence and Prospect of General Knowledge Transfer between Any Pretrained Model [74.62272538148245]
  We show that for arbitrary pairings of pretrained models, one model extracts significant data context unavailable in the other. We investigate if it is possible to transfer such "complementary" knowledge from one model to another without performance degradation.
  arXiv  Detail & Related papers  (2023-10-26T17:59:46Z)
• Fast and Accurate Transferability Measurement by Evaluating Intra-class Feature Variance [20.732095457775138]
  Transferability measurement is to quantify how transferable is a pre-trained model learned on a source task to a target task. We propose TMI (TRANSFERABILITY MEASUREMENT WITH INTRA-CLASS FEATURE VARIANCE), a fast and accurate algorithm to measure transferability.
  arXiv  Detail & Related papers  (2023-08-11T07:50:40Z)
• SynBench: Task-Agnostic Benchmarking of Pretrained Representations using Synthetic Data [78.21197488065177]
  Recent success in fine-tuning large models, that are pretrained on broad data at scale, on downstream tasks has led to a significant paradigm shift in deep learning. This paper proposes a new task-agnostic framework, textitSynBench, to measure the quality of pretrained representations using synthetic data.
  arXiv  Detail & Related papers  (2022-10-06T15:25:00Z)
• Part-Based Models Improve Adversarial Robustness [57.699029966800644]
  We show that combining human prior knowledge with end-to-end learning can improve the robustness of deep neural networks. Our model combines a part segmentation model with a tiny classifier and is trained end-to-end to simultaneously segment objects into parts. Our experiments indicate that these models also reduce texture bias and yield better robustness against common corruptions and spurious correlations.
  arXiv  Detail & Related papers  (2022-09-15T15:41:47Z)
• When does Bias Transfer in Transfer Learning? [89.22641454588278]
  Using transfer learning to adapt a pre-trained "source model" to a downstream "target task" can dramatically increase performance with seemingly no downside. We demonstrate that there can exist a downside after all: bias transfer, or the tendency for biases of the source model to persist even after adapting the model to the target class.
  arXiv  Detail & Related papers  (2022-07-06T17:58:07Z)
• Revisiting the Updates of a Pre-trained Model for Few-shot Learning [11.871523410051527]
  We compare the two popular updating methods, fine-tuning and linear probing. We find that fine-tuning is better than linear probing as the number of samples increases.
  arXiv  Detail & Related papers  (2022-05-13T08:47:06Z)
• Diverse Imagenet Models Transfer Better [10.6046072921331]
  We show that high diversity of features learnt by a model promotes transferability jointly with Imagenet accuracy. We propose a method that combines self-supervised and supervised pretraining to generate models with both high diversity and high accuracy.
  arXiv  Detail & Related papers  (2022-04-19T21:26:58Z)
• How Well Do Sparse Imagenet Models Transfer? [75.98123173154605]
  Transfer learning is a classic paradigm by which models pretrained on large "upstream" datasets are adapted to yield good results on "downstream" datasets. In this work, we perform an in-depth investigation of this phenomenon in the context of convolutional neural networks (CNNs) trained on the ImageNet dataset. We show that sparse models can match or even outperform the transfer performance of dense models, even at high sparsities.
  arXiv  Detail & Related papers  (2021-11-26T11:58:51Z)
• Deep Ensembles for Low-Data Transfer Learning [21.578470914935938]
  We study different ways of creating ensembles from pre-trained models. We show that the nature of pre-training itself is a performant source of diversity. We propose a practical algorithm that efficiently identifies a subset of pre-trained models for any downstream dataset.
  arXiv  Detail & Related papers  (2020-10-14T07:59:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.