What is being transferred in transfer learning?
- URL: http://arxiv.org/abs/2008.11687v2
- Date: Thu, 14 Jan 2021 20:32:39 GMT
- Title: What is being transferred in transfer learning?
- Authors: Behnam Neyshabur and Hanie Sedghi and Chiyuan Zhang
- Abstract summary: We show that when training from pre-trained weights, the model stays in the same basin in the loss landscape.
We present that when training from pre-trained weights, the model stays in the same basin in the loss landscape and different instances of such model are similar in feature space and close in parameter space.
- Score: 51.6991244438545
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: One desired capability for machines is the ability to transfer their
knowledge of one domain to another where data is (usually) scarce. Despite
ample adaptation of transfer learning in various deep learning applications, we
yet do not understand what enables a successful transfer and which part of the
network is responsible for that. In this paper, we provide new tools and
analyses to address these fundamental questions. Through a series of analyses
on transferring to block-shuffled images, we separate the effect of feature
reuse from learning low-level statistics of data and show that some benefit of
transfer learning comes from the latter. We present that when training from
pre-trained weights, the model stays in the same basin in the loss landscape
and different instances of such model are similar in feature space and close in
parameter space.
Related papers
- A Data-Based Perspective on Transfer Learning [76.30206800557411]
We take a closer look at the role of the source dataset's composition in transfer learning.
Our framework gives rise to new capabilities such as pinpointing transfer learning brittleness.
arXiv Detail & Related papers (2022-07-12T17:58:28Z) - Intra-domain and cross-domain transfer learning for time series data --
How transferable are the features? [0.0]
This study aims to assess how transferable are the features between different domains of time series data.
The effects of transfer learning are observed in terms of predictive performance of the models and their convergence rate during training.
arXiv Detail & Related papers (2022-01-12T12:55:21Z) - Disentangling Transfer and Interference in Multi-Domain Learning [53.34444188552444]
We study the conditions where interference and knowledge transfer occur in multi-domain learning.
We propose new metrics disentangling interference and transfer and set up experimental protocols.
We demonstrate our findings on the CIFAR-100, MiniPlaces, and Tiny-ImageNet datasets.
arXiv Detail & Related papers (2021-07-02T01:30:36Z) - A Concise Review of Transfer Learning [1.5771347525430772]
Transfer learning aims to boost the performance of a target learner by applying another related source data.
Traditional machine learning and data mining techniques assume that the training and testing data lie from the same feature space and distribution.
arXiv Detail & Related papers (2021-04-05T20:34:55Z) - Factors of Influence for Transfer Learning across Diverse Appearance
Domains and Task Types [50.1843146606122]
A simple form of transfer learning is common in current state-of-the-art computer vision models.
Previous systematic studies of transfer learning have been limited and the circumstances in which it is expected to work are not fully understood.
In this paper we carry out an extensive experimental exploration of transfer learning across vastly different image domains.
arXiv Detail & Related papers (2021-03-24T16:24:20Z) - Unsupervised Transfer Learning for Spatiotemporal Predictive Networks [90.67309545798224]
We study how to transfer knowledge from a zoo of unsupervisedly learned models towards another network.
Our motivation is that models are expected to understand complex dynamics from different sources.
Our approach yields significant improvements on three benchmarks fortemporal prediction, and benefits the target even from less relevant ones.
arXiv Detail & Related papers (2020-09-24T15:40:55Z) - Adversarially-Trained Deep Nets Transfer Better: Illustration on Image
Classification [53.735029033681435]
Transfer learning is a powerful methodology for adapting pre-trained deep neural networks on image recognition tasks to new domains.
In this work, we demonstrate that adversarially-trained models transfer better than non-adversarially-trained models.
arXiv Detail & Related papers (2020-07-11T22:48:42Z) - Limits of Transfer Learning [0.0]
We show the need to carefully select which sets of information to transfer and the need for dependence between transferred information and target problems.
These results build on the algorithmic search framework for machine learning, allowing the results to apply to a wide range of learning problems using transfer.
arXiv Detail & Related papers (2020-06-23T01:48:23Z) - Exploring the Efficacy of Transfer Learning in Mining Image-Based
Software Artifacts [1.5285292154680243]
Transfer learning allows us to train deep architectures requiring a large number of learned parameters, even if the amount of available data is limited.
Here we explore the applicability of transfer learning utilizing models pre-trained on non-software engineering data applied to the problem of classifying software diagrams.
arXiv Detail & Related papers (2020-03-03T16:41:45Z) - Inter- and Intra-domain Knowledge Transfer for Related Tasks in Deep
Character Recognition [2.320417845168326]
Pre-training a deep neural network on the ImageNet dataset is a common practice for training deep learning models.
The technique of pre-training on one task and then retraining on a new one is called transfer learning.
In this paper we analyse the effectiveness of using deep transfer learning for character recognition tasks.
arXiv Detail & Related papers (2020-01-02T14:18:25Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.