Robust Tickets Can Transfer Better: Drawing More Transferable Subnetworks in Transfer Learning

• URL: http://arxiv.org/abs/2304.11834v2
• Date: Fri, 7 Jul 2023 03:17:46 GMT
• Title: Robust Tickets Can Transfer Better: Drawing More Transferable Subnetworks in Transfer Learning
• Authors: Yonggan Fu, Ye Yuan, Shang Wu, Jiayi Yuan, Yingyan Lin
• Abstract summary: Transfer learning leverages feature representations of deep neural networks (DNNs) pretrained on source tasks with rich data to empower finetuning on downstream tasks. We propose a new transfer learning pipeline, which leverages our finding that robust tickets can transfer better, i.e.,works drawn with properly induced adversarial robustness can win better transferability over vanilla lottery ticketworks.
• Score: 25.310066345466396
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Transfer learning leverages feature representations of deep neural networks (DNNs) pretrained on source tasks with rich data to empower effective finetuning on downstream tasks. However, the pretrained models are often prohibitively large for delivering generalizable representations, which limits their deployment on edge devices with constrained resources. To close this gap, we propose a new transfer learning pipeline, which leverages our finding that robust tickets can transfer better, i.e., subnetworks drawn with properly induced adversarial robustness can win better transferability over vanilla lottery ticket subnetworks. Extensive experiments and ablation studies validate that our proposed transfer learning pipeline can achieve enhanced accuracy-sparsity trade-offs across both diverse downstream tasks and sparsity patterns, further enriching the lottery ticket hypothesis.

Related papers

• Risk of Transfer Learning and its Applications in Finance [2.966069495345018]
  We propose a novel concept of transfer risk and analyze its properties to evaluate transferability of transfer learning. Numerical results demonstrate a strong correlation between transfer risk and overall transfer learning performance.
  arXiv  Detail & Related papers  (2023-11-06T17:23:54Z)
• Robust Transfer Learning with Unreliable Source Data [13.276850367115333]
  We introduce a novel quantity called the ''ambiguity level'' that measures the discrepancy between the target and source regression functions. We propose a simple transfer learning procedure, and establish a general theorem that shows how this new quantity is related to the transferability of learning.
  arXiv  Detail & Related papers  (2023-10-06T21:50:21Z)
• Data-Efficient Double-Win Lottery Tickets from Robust Pre-training [129.85939347733387]
  We introduce Double-Win Lottery Tickets, in which a subnetwork from a pre-trained model can be independently transferred on diverse downstream tasks. We find that robust pre-training tends to craft sparser double-win lottery tickets with superior performance over the standard counterparts.
  arXiv  Detail & Related papers  (2022-06-09T20:52:50Z)
• Learning to Win Lottery Tickets in BERT Transfer via Task-agnostic Mask Training [55.43088293183165]
  Recent studies show that pre-trained language models (PLMs) like BERT contain matchingworks that have similar transfer learning performance as the original PLM. In this paper, we find that the BERTworks have even more potential than these studies have shown. We train binary masks over model weights on the pre-training tasks, with the aim of preserving the universal transferability of the subnetwork.
  arXiv  Detail & Related papers  (2022-04-24T08:42:47Z)
• Self-Supervised Knowledge Transfer via Loosely Supervised Auxiliary Tasks [24.041268664220294]
  knowledge transfer using convolutional neural networks (CNNs) can help efficiently train a CNN with fewer parameters or maximize the generalization performance under limited supervision. We propose a simple yet powerful knowledge transfer methodology without any restrictions regarding the network structure or dataset used. We devise a training methodology that transfers previously learned knowledge to the current training process as an auxiliary task for the target task through self-supervision using a soft label.
  arXiv  Detail & Related papers  (2021-10-25T07:18:26Z)
• Adversarial Training Helps Transfer Learning via Better Representations [17.497590668804055]
  Transfer learning aims to leverage models pre-trained on source data to efficiently adapt to target setting. Recent works empirically demonstrate that adversarial training in the source data can improve the ability of models to transfer to new domains. We show that adversarial training in the source data generates provably better representations, so fine-tuning on top of this representation leads to a more accurate predictor of the target data.
  arXiv  Detail & Related papers  (2021-06-18T15:41:07Z)
• Frustratingly Easy Transferability Estimation [64.42879325144439]
  We propose a simple, efficient, and effective transferability measure named TransRate. TransRate measures the transferability as the mutual information between the features of target examples extracted by a pre-trained model and labels of them. Despite its extraordinary simplicity in 10 lines of codes, TransRate performs remarkably well in extensive evaluations on 22 pre-trained models and 16 downstream tasks.
  arXiv  Detail & Related papers  (2021-06-17T10:27:52Z)
• Unsupervised Transfer Learning for Spatiotemporal Predictive Networks [90.67309545798224]
  We study how to transfer knowledge from a zoo of unsupervisedly learned models towards another network. Our motivation is that models are expected to understand complex dynamics from different sources. Our approach yields significant improvements on three benchmarks fortemporal prediction, and benefits the target even from less relevant ones.
  arXiv  Detail & Related papers  (2020-09-24T15:40:55Z)
• Adversarial Training Reduces Information and Improves Transferability [81.59364510580738]
  Recent results show that features of adversarially trained networks for classification, in addition to being robust, enable desirable properties such as invertibility. We show that the Adversarial Training can improve linear transferability to new tasks, from which arises a new trade-off between transferability of representations and accuracy on the source task.
  arXiv  Detail & Related papers  (2020-07-22T08:30:16Z)
• Adversarially-Trained Deep Nets Transfer Better: Illustration on Image Classification [53.735029033681435]
  Transfer learning is a powerful methodology for adapting pre-trained deep neural networks on image recognition tasks to new domains. In this work, we demonstrate that adversarially-trained models transfer better than non-adversarially-trained models.
  arXiv  Detail & Related papers  (2020-07-11T22:48:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.