Transfer Learning in Infinite Width Feature Learning Networks

• URL: http://arxiv.org/abs/2507.04448v1
• Date: Sun, 06 Jul 2025 16:14:43 GMT
• Title: Transfer Learning in Infinite Width Feature Learning Networks
• Authors: Clarissa Lauditi, Blake Bordelon, Cengiz Pehlevan,
• Abstract summary: We develop a theory of transfer learning in infinitely wide neural networks where both the pretraining (source) and downstream (target) task can operate in a feature learning regime.<n>We analyze both the Bayesian framework, where learning is described by a posterior distribution over the weights, and gradient flow training of randomly gradient networks trained with weight decay.<n>The summary statistics of these theories are adapted feature kernels which, after transfer learning, depend on data and labels from both source and target tasks.
• Score: 35.95321041944522
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We develop a theory of transfer learning in infinitely wide neural networks where both the pretraining (source) and downstream (target) task can operate in a feature learning regime. We analyze both the Bayesian framework, where learning is described by a posterior distribution over the weights, and gradient flow training of randomly initialized networks trained with weight decay. Both settings track how representations evolve in both source and target tasks. The summary statistics of these theories are adapted feature kernels which, after transfer learning, depend on data and labels from both source and target tasks. Reuse of features during transfer learning is controlled by an elastic weight coupling which controls the reliance of the network on features learned during training on the source task. We apply our theory to linear and polynomial regression tasks as well as real datasets. Our theory and experiments reveal interesting interplays between elastic weight coupling, feature learning strength, dataset size, and source and target task alignment on the utility of transfer learning.

Related papers

• Features are fate: a theory of transfer learning in high-dimensional regression [23.840251319669907]
  We show that when the target task is well represented by the feature space of the pre-trained model, transfer learning outperforms training from scratch. For this model, we establish rigorously that when the feature space overlap between the source and target tasks is sufficiently strong, both linear transfer and fine-tuning improve performance.
  arXiv  Detail & Related papers  (2024-10-10T17:58:26Z)
• Diffusion-Based Neural Network Weights Generation [80.89706112736353]
  D2NWG is a diffusion-based neural network weights generation technique that efficiently produces high-performing weights for transfer learning. Our method extends generative hyper-representation learning to recast the latent diffusion paradigm for neural network weights generation. Our approach is scalable to large architectures such as large language models (LLMs), overcoming the limitations of current parameter generation techniques.
  arXiv  Detail & Related papers  (2024-02-28T08:34:23Z)
• IF2Net: Innately Forgetting-Free Networks for Continual Learning [49.57495829364827]
  Continual learning can incrementally absorb new concepts without interfering with previously learned knowledge. Motivated by the characteristics of neural networks, we investigated how to design an Innately Forgetting-Free Network (IF2Net) IF2Net allows a single network to inherently learn unlimited mapping rules without telling task identities at test time.
  arXiv  Detail & Related papers  (2023-06-18T05:26:49Z)
• An information-Theoretic Approach to Semi-supervised Transfer Learning [33.89602092349131]
  Transfer learning allows propagating information from one "source dataset" to another "target dataset" discrepancies between the underlying distributions of the source and target data are commonplace. We suggest novel information-theoretic approaches for the analysis of the performance of deep neural networks in the context of transfer learning.
  arXiv  Detail & Related papers  (2023-06-11T17:45:46Z)
• Optimal transfer protocol by incremental layer defrosting [66.76153955485584]
  Transfer learning is a powerful tool enabling model training with limited amounts of data. The simplest transfer learning protocol is based on freezing" the feature-extractor layers of a network pre-trained on a data-rich source task. We show that this protocol is often sub-optimal and the largest performance gain may be achieved when smaller portions of the pre-trained network are kept frozen.
  arXiv  Detail & Related papers  (2023-03-02T17:32:11Z)
• Adaptive Parameterization of Deep Learning Models for Federated Learning [85.82002651944254]
  Federated Learning offers a way to train deep neural networks in a distributed fashion. It incurs a communication overhead as the model parameters or gradients need to be exchanged regularly during training. In this paper, we propose to utilise parallel Adapters for Federated Learning.
  arXiv  Detail & Related papers  (2023-02-06T17:30:33Z)
• Frozen Overparameterization: A Double Descent Perspective on Transfer Learning of Deep Neural Networks [27.17697714584768]
  We study the generalization behavior of transfer learning of deep neural networks (DNNs) We show that the test error evolution during the target training has a more significant double descent effect when the target training dataset is sufficiently large. Also, we show that the double descent phenomenon may make a transfer from a less related source task better than a transfer from a more related source task.
  arXiv  Detail & Related papers  (2022-11-20T20:26:23Z)
• Meta-learning Transferable Representations with a Single Target Domain [46.83481356352768]
  Fine-tuning and joint training do not always improve accuracy on downstream tasks. We propose Meta Representation Learning (MeRLin) to learn transferable features. MeRLin empirically outperforms previous state-of-the-art transfer learning algorithms on various real-world vision and NLP transfer learning benchmarks.
  arXiv  Detail & Related papers  (2020-11-03T01:57:37Z)
• Unsupervised Transfer Learning for Spatiotemporal Predictive Networks [90.67309545798224]
  We study how to transfer knowledge from a zoo of unsupervisedly learned models towards another network. Our motivation is that models are expected to understand complex dynamics from different sources. Our approach yields significant improvements on three benchmarks fortemporal prediction, and benefits the target even from less relevant ones.
  arXiv  Detail & Related papers  (2020-09-24T15:40:55Z)
• What is being transferred in transfer learning? [51.6991244438545]
  We show that when training from pre-trained weights, the model stays in the same basin in the loss landscape. We present that when training from pre-trained weights, the model stays in the same basin in the loss landscape and different instances of such model are similar in feature space and close in parameter space.
  arXiv  Detail & Related papers  (2020-08-26T17:23:40Z)
• Minimax Lower Bounds for Transfer Learning with Linear and One-hidden Layer Neural Networks [27.44348371795822]
  We develop a statistical minimax framework to characterize the limits of transfer learning. We derive a lower-bound for the target generalization error achievable by any algorithm as a function of the number of labeled source and target data.
  arXiv  Detail & Related papers  (2020-06-16T22:49:26Z)
• Inter- and Intra-domain Knowledge Transfer for Related Tasks in Deep Character Recognition [2.320417845168326]
  Pre-training a deep neural network on the ImageNet dataset is a common practice for training deep learning models. The technique of pre-training on one task and then retraining on a new one is called transfer learning. In this paper we analyse the effectiveness of using deep transfer learning for character recognition tasks.
  arXiv  Detail & Related papers  (2020-01-02T14:18:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.