Sample Complexity of Transfer Learning: An Optimal Transport Approach

• URL: http://arxiv.org/abs/2605.20545v1
• Date: Tue, 19 May 2026 22:39:03 GMT
• Title: Sample Complexity of Transfer Learning: An Optimal Transport Approach
• Abstract summary: In this work, we rigorously analyze the potential benefit of transfer learning in terms of sample efficiency.<n>We find that when the data dimension $d$ is higher than $3$, the sample complexity for transfer learning is $O(m-(+1)/d)$.<n>Our finding theoretically supports a better sample efficiency for transfer learning, when the target task is optimizing over a family of not-so-smooth models.
• Score: 18.537610792819176
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Transfer learning is an essential technique for many machine learning/AI models of complex structures such as large language models and generative AI. The essence of transfer learning is to leverage knowledge from resolved source tasks for a new target task, especially when the sample size $m$ of the training data for the latter is low. In this work, we rigorously analyze the potential benefit of transfer learning in terms of sample efficiency. Specifically, taking an optimal transport viewpoint of transfer learning, we find that when the data dimension $d$ is higher than $3$, the sample complexity for transfer learning is $O(m^{-(α+1)/d})$, with $α$ indicating the smoothness of the data distribution, as opposed to the $O(m^{-p/d})$ sample complexity for direct learning with $p$ indicating the smoothness of the optimal target model. Our finding theoretically supports a better sample efficiency for transfer learning, when the target task is optimizing over a family of not-so-smooth models (i.e., highly complex networks with the possible use of non-smooth activation functions). Using image classification as an example, we numerically demonstrate the sample efficiency for transfer learning, that is, in the data hungry regime, the model performance can be significantly improved by transfer learning.

Related papers

• Learning to Unlearn: Machine Unlearning via Learning the Unlearning Behaviors [17.009892363181567]
  We introduce the first learning-based model-agnostic approach, Learning-to-UnLearn (L2UL)<n>Our core insight is to shift from manually designing $U$ to learning the unlearning behaviors from a distribution perspective.<n>Our experimental results demonstrate that the accuracy achieved by L2UL is comparable to that of retraining.
  arXiv  Detail & Related papers  (2026-08-17T15:18:34Z)
• Learning Multi-Indicator Weights for Data Selection: A Joint Task-Model Adaptation Framework with Efficient Proxies [50.39041754816285]
  We propose a framework for learning multi-validation weights that jointly adapts data selection to both the downstream task and the specific model.<n>Our approach achieves performance comparable to or exceeding full-dataset tuning while using only 30% of the training samples on GSM8K.
  arXiv  Detail & Related papers  (2026-05-10T17:30:16Z)
• When Does Visual Prompting Outperform Linear Probing for Vision-Language Models? A Likelihood Perspective [57.05315507519704]
  We propose a log-likelihood ratio (LLR) approach to analyze the comparative benefits of visual prompting and linear probing. Our measure attains up to a 100-fold reduction in run time compared to full training, while achieving prediction accuracies up to 91%.
  arXiv  Detail & Related papers  (2024-09-03T12:03:45Z)
• Diffusion-Based Neural Network Weights Generation [80.89706112736353]
  D2NWG is a diffusion-based neural network weights generation technique that efficiently produces high-performing weights for transfer learning. Our method extends generative hyper-representation learning to recast the latent diffusion paradigm for neural network weights generation. Our approach is scalable to large architectures such as large language models (LLMs), overcoming the limitations of current parameter generation techniques.
  arXiv  Detail & Related papers  (2024-02-28T08:34:23Z)
• Fast Exact Unlearning for In-Context Learning Data for LLMs [30.06631665962119]
  We show that for large language models we can efficiently unlearn "fine-tuning data"<n>We show that accurate in-context learning can be done with quantized k-means, which allows for effectively constant time unlearning operations.
  arXiv  Detail & Related papers  (2024-02-01T16:43:04Z)
• Optimal transfer protocol by incremental layer defrosting [66.76153955485584]
  Transfer learning is a powerful tool enabling model training with limited amounts of data. The simplest transfer learning protocol is based on freezing" the feature-extractor layers of a network pre-trained on a data-rich source task. We show that this protocol is often sub-optimal and the largest performance gain may be achieved when smaller portions of the pre-trained network are kept frozen.
  arXiv  Detail & Related papers  (2023-03-02T17:32:11Z)
• Deep invariant networks with differentiable augmentation layers [87.22033101185201]
  Methods for learning data augmentation policies require held-out data and are based on bilevel optimization problems. We show that our approach is easier and faster to train than modern automatic data augmentation techniques.
  arXiv  Detail & Related papers  (2022-02-04T14:12:31Z)
• A Bayesian Approach to (Online) Transfer Learning: Theory and Algorithms [6.193838300896449]
  We study transfer learning from a Bayesian perspective, where a parametric statistical model is used. Specifically, we study three variants of transfer learning problems, instantaneous, online, and time-variant transfer learning. For each problem, we define an appropriate objective function, and provide either exact expressions or upper bounds on the learning performance. Examples show that the derived bounds are accurate even for small sample sizes.
  arXiv  Detail & Related papers  (2021-09-03T08:43:29Z)
• Fractional Transfer Learning for Deep Model-Based Reinforcement Learning [0.966840768820136]
  Reinforcement learning (RL) is well known for requiring large amounts of data in order for RL agents to learn to perform complex tasks. Recent progress in model-based RL allows agents to be much more data-efficient. We present a simple alternative approach: fractional transfer learning.
  arXiv  Detail & Related papers  (2021-08-14T12:44:42Z)
• Probing transfer learning with a model of synthetic correlated datasets [11.53207294639557]
  Transfer learning can significantly improve the sample efficiency of neural networks. We re-think a solvable model of synthetic data as a framework for modeling correlation between data-sets. We show that our model can capture a range of salient features of transfer learning with real data.
  arXiv  Detail & Related papers  (2021-06-09T22:15:41Z)
• Cost-Effective Federated Learning Design [37.16466118235272]
  Federated learning (FL) is a distributed learning paradigm that enables a large number of devices to collaboratively learn a model without sharing their raw data. Despite its efficiency and effectiveness, the iterative on-device learning process incurs a considerable cost in terms of learning time and energy consumption. We analyze how to design adaptive FL that optimally chooses essential control variables to minimize the total cost while ensuring convergence.
  arXiv  Detail & Related papers  (2020-12-15T14:45:11Z)
• XMixup: Efficient Transfer Learning with Auxiliary Samples by Cross-domain Mixup [60.07531696857743]
  Cross-domain Mixup (XMixup) improves the multitask paradigm for deep transfer learning. XMixup selects the auxiliary samples from the source dataset and augments training samples via the simple mixup strategy. Experiment results show that XMixup improves the accuracy by 1.9% on average.
  arXiv  Detail & Related papers  (2020-07-20T16:42:29Z)
• Parameter-Efficient Transfer from Sequential Behaviors for User Modeling and Recommendation [111.44445634272235]
  In this paper, we develop a parameter efficient transfer learning architecture, termed as PeterRec. PeterRec allows the pre-trained parameters to remain unaltered during fine-tuning by injecting a series of re-learned neural networks. We perform extensive experimental ablation to show the effectiveness of the learned user representation in five downstream tasks.
  arXiv  Detail & Related papers  (2020-01-13T14:09:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.