Limits of Transfer Learning

• URL: http://arxiv.org/abs/2006.12694v1
• Date: Tue, 23 Jun 2020 01:48:23 GMT
• Title: Limits of Transfer Learning
• Authors: Jake Williams, Abel Tadesse, Tyler Sam, Huey Sun, George D. Montanez
• Abstract summary: We show the need to carefully select which sets of information to transfer and the need for dependence between transferred information and target problems. These results build on the algorithmic search framework for machine learning, allowing the results to apply to a wide range of learning problems using transfer.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Transfer learning involves taking information and insight from one problem domain and applying it to a new problem domain. Although widely used in practice, theory for transfer learning remains less well-developed. To address this, we prove several novel results related to transfer learning, showing the need to carefully select which sets of information to transfer and the need for dependence between transferred information and target problems. Furthermore, we prove how the degree of probabilistic change in an algorithm using transfer learning places an upper bound on the amount of improvement possible. These results build on the algorithmic search framework for machine learning, allowing the results to apply to a wide range of learning problems using transfer.

Related papers

• Bayesian Transfer Learning [13.983016833412307]
  "Transfer learning" seeks to improve inference and/or predictive accuracy on a domain of interest by leveraging data from related domains. This article highlights Bayesian approaches to transfer learning, which have received relatively limited attention despite their innate compatibility with the notion of drawing upon prior knowledge to guide new learning tasks. We discuss how these methods address the problem of finding the optimal information to transfer between domains, which is a central question in transfer learning.
  arXiv  Detail & Related papers  (2023-12-20T23:38:17Z)
• Feasibility of Transfer Learning: A Mathematical Framework [4.530876736231948]
  It begins by establishing the necessary mathematical concepts and constructing a mathematical framework for transfer learning. It then identifies and formulates the three-step transfer learning procedure as an optimization problem, allowing for the resolution of the feasibility issue.
  arXiv  Detail & Related papers  (2023-05-22T12:44:38Z)
• Transferability in Deep Learning: A Survey [80.67296873915176]
  The ability to acquire and reuse knowledge is known as transferability in deep learning. We present this survey to connect different isolated areas in deep learning with their relation to transferability. We implement a benchmark and an open-source library, enabling a fair evaluation of deep learning methods in terms of transferability.
  arXiv  Detail & Related papers  (2022-01-15T15:03:17Z)
• A Bayesian Approach to (Online) Transfer Learning: Theory and Algorithms [6.193838300896449]
  We study transfer learning from a Bayesian perspective, where a parametric statistical model is used. Specifically, we study three variants of transfer learning problems, instantaneous, online, and time-variant transfer learning. For each problem, we define an appropriate objective function, and provide either exact expressions or upper bounds on the learning performance. Examples show that the derived bounds are accurate even for small sample sizes.
  arXiv  Detail & Related papers  (2021-09-03T08:43:29Z)
• Disentangling Transfer and Interference in Multi-Domain Learning [53.34444188552444]
  We study the conditions where interference and knowledge transfer occur in multi-domain learning. We propose new metrics disentangling interference and transfer and set up experimental protocols. We demonstrate our findings on the CIFAR-100, MiniPlaces, and Tiny-ImageNet datasets.
  arXiv  Detail & Related papers  (2021-07-02T01:30:36Z)
• Online Transfer Learning: Negative Transfer and Effect of Prior Knowledge [6.193838300896449]
  We study the online transfer learning problems where the source samples are given in an offline way while the target samples arrive sequentially. We define the expected regret of the online transfer learning problem and provide upper bounds on the regret using information-theoretic quantities. Examples show that the derived bounds are accurate even for small sample sizes.
  arXiv  Detail & Related papers  (2021-05-04T12:12:14Z)
• Language Model is All You Need: Natural Language Understanding as Question Answering [75.26830242165742]
  We study the use of a specific family of transfer learning, where the target domain is mapped to the source domain. We show that our approach could reduce the amount of required data for the same performance by up to a factor of 10.
  arXiv  Detail & Related papers  (2020-11-05T18:31:22Z)
• Transfer Learning in Deep Reinforcement Learning: A Survey [64.36174156782333]
  Reinforcement learning is a learning paradigm for solving sequential decision-making problems. Recent years have witnessed remarkable progress in reinforcement learning upon the fast development of deep neural networks. transfer learning has arisen to tackle various challenges faced by reinforcement learning.
  arXiv  Detail & Related papers  (2020-09-16T18:38:54Z)
• What is being transferred in transfer learning? [51.6991244438545]
  We show that when training from pre-trained weights, the model stays in the same basin in the loss landscape. We present that when training from pre-trained weights, the model stays in the same basin in the loss landscape and different instances of such model are similar in feature space and close in parameter space.
  arXiv  Detail & Related papers  (2020-08-26T17:23:40Z)
• Unsupervised Transfer Learning with Self-Supervised Remedy [60.315835711438936]
  Generalising deep networks to novel domains without manual labels is challenging to deep learning. Pre-learned knowledge does not transfer well without making strong assumptions about the learned and the novel domains. In this work, we aim to learn a discriminative latent space of the unlabelled target data in a novel domain by knowledge transfer from labelled related domains.
  arXiv  Detail & Related papers  (2020-06-08T16:42:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.