A High-Dimensional Statistical Method for Optimizing Transfer Quantities in Multi-Source Transfer Learning

• URL: http://arxiv.org/abs/2502.04242v4
• Date: Tue, 28 Oct 2025 06:15:39 GMT
• Title: A High-Dimensional Statistical Method for Optimizing Transfer Quantities in Multi-Source Transfer Learning
• Authors: Qingyue Zhang, Haohao Fu, Guanbo Huang, Yaoyuan Liang, Chang Chu, Tianren Peng, Yanru Wu, Qi Li, Yang Li, Shao-Lun Huang,
• Abstract summary: We propose a theoretical framework that answers the question: what is the optimal quantity of source samples needed from each source task to jointly train the target model?<n>Specifically, we introduce a generalization error measure based on K-L divergence, and minimize it based on high-dimensional statistical analysis to determine the optimal transfer quantity for each source task.<n>We develop an architecture-agnostic and data-efficient algorithm OTQMS to implement our theoretical results for target model training in multi-source transfer learning.
• Score: 19.59729349421865
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multi-source transfer learning provides an effective solution to data scarcity in real-world supervised learning scenarios by leveraging multiple source tasks. In this field, existing works typically use all available samples from sources in training, which constrains their training efficiency and may lead to suboptimal results. To address this, we propose a theoretical framework that answers the question: what is the optimal quantity of source samples needed from each source task to jointly train the target model? Specifically, we introduce a generalization error measure based on K-L divergence, and minimize it based on high-dimensional statistical analysis to determine the optimal transfer quantity for each source task. Additionally, we develop an architecture-agnostic and data-efficient algorithm OTQMS to implement our theoretical results for target model training in multi-source transfer learning. Experimental studies on diverse architectures and two real-world benchmark datasets show that our proposed algorithm significantly outperforms state-of-the-art approaches in both accuracy and data efficiency. The code and supplementary materials are available in https://github.com/zqy0126/OTQMS.

Related papers

• Unified Optimization of Source Weights and Transfer Quantities in Multi-Source Transfer Learning: An Asymptotic Framework [15.078086239740342]
  We propose a theoretical framework, Unified Optimization of Weights and Quantities (UOWQ), which formulates multi-source transfer learning as a parameter estimation problem.<n>We prove that using all available source samples is always optimal once the weights are properly adjusted.<n>Building on the theoretical results, we further propose practical algorithms for both multi-source transfer learning and multi-task learning settings.
  arXiv  Detail & Related papers  (2026-01-15T18:46:54Z)
• SPaRFT: Self-Paced Reinforcement Fine-Tuning for Large Language Models [51.74498855100541]
  Large language models (LLMs) have shown strong reasoning capabilities when fine-tuned with reinforcement learning (RL)<n>We propose textbfSPaRFT, a self-paced learning framework that enables efficient learning based on the capability of the model being trained.
  arXiv  Detail & Related papers  (2025-08-07T03:50:48Z)
• Data Uniformity Improves Training Efficiency and More, with a Convergence Framework Beyond the NTK Regime [9.749891245059596]
  We demonstrate that selecting more uniformly distributed data can improve training efficiency while enhancing performance.<n>Specifically, we establish that more uniform (less biased) distribution leads to a larger minimum pairwise distance between data points.<n>We theoretically show that the approximation error of neural networks decreases as $h_min$ increases.
  arXiv  Detail & Related papers  (2025-06-30T17:58:30Z)
• BeST -- A Novel Source Selection Metric for Transfer Learning [35.32994166809785]
  We develop a novel task-similarity metric (BeST) to identify the most transferrable source(s) for a given task.<n>Our metric can provide significant computational savings for transfer learning from a selection of a large number of possible source models.
  arXiv  Detail & Related papers  (2025-01-19T03:58:05Z)
• Empowering Large Language Models in Wireless Communication: A Novel Dataset and Fine-Tuning Framework [81.29965270493238]
  We develop a specialized dataset aimed at enhancing the evaluation and fine-tuning of large language models (LLMs) for wireless communication applications.<n>The dataset includes a diverse set of multi-hop questions, including true/false and multiple-choice types, spanning varying difficulty levels from easy to hard.<n>We introduce a Pointwise V-Information (PVI) based fine-tuning method, providing a detailed theoretical analysis and justification for its use in quantifying the information content of training data.
  arXiv  Detail & Related papers  (2025-01-16T16:19:53Z)
• Transfer Learning with Reconstruction Loss [12.906500431427716]
  This paper proposes a novel approach for model training by adding into the model an additional reconstruction stage associated with a new reconstruction loss. The proposed approach encourages the learned features to be general and transferable, and therefore can be readily used for efficient transfer learning. For numerical simulations, three applications are studied: transfer learning on classifying MNIST handwritten digits, the device-to-device wireless network power allocation, and the multiple-input-single-output network downlink beamforming and localization.
  arXiv  Detail & Related papers  (2024-03-31T00:22:36Z)
• Long-Tailed Recognition on Binary Networks by Calibrating A Pre-trained Model [18.58663937035378]
  We address the combined challenge of learning long-tailed distributions using highly resource-efficient binary neural networks as backbones. We propose a calibrate-and-distill framework that uses off-the-shelf pretrained full-precision models trained on balanced datasets to use as teachers for distillation. To better generalize to various datasets, we propose a novel adversarial balancing among the terms in the objective function and an efficient multiresolution learning scheme.
  arXiv  Detail & Related papers  (2024-03-30T08:37:19Z)
• Diffusion-Based Neural Network Weights Generation [80.89706112736353]
  D2NWG is a diffusion-based neural network weights generation technique that efficiently produces high-performing weights for transfer learning. Our method extends generative hyper-representation learning to recast the latent diffusion paradigm for neural network weights generation. Our approach is scalable to large architectures such as large language models (LLMs), overcoming the limitations of current parameter generation techniques.
  arXiv  Detail & Related papers  (2024-02-28T08:34:23Z)
• How to Train Data-Efficient LLMs [56.41105687693619]
  We study data-efficient approaches for pre-training language models (LLMs) We find that Ask-LLM and Density sampling are the best methods in their respective categories. In our comparison of 19 samplers, involving hundreds of evaluation tasks and pre-training runs, we find that Ask-LLM and Density are the best methods in their respective categories.
  arXiv  Detail & Related papers  (2024-02-15T02:27:57Z)
• H-ensemble: An Information Theoretic Approach to Reliable Few-Shot Multi-Source-Free Transfer [4.328706834250445]
  We propose a framework named H-ensemble, which learns the optimal linear combination of source models for the target task. Compared to previous works, H-ensemble is characterized by: 1) its adaptability to a novel MSF setting for few-shot target tasks, 2) theoretical reliability, 3) a lightweight structure easy to interpret and adapt. We show that the H-ensemble can successfully learn the optimal task ensemble, as well as outperform prior arts.
  arXiv  Detail & Related papers  (2023-12-19T17:39:34Z)
• Efficient Training of Multi-task Neural Solver for Combinatorial Optimization [23.694457372640912]
  We propose a general and efficient training paradigm to deliver a unified multi-task neural solver. Our method significantly enhances overall performance, regardless of whether it is within constrained training budgets. Our method also achieved the best results compared to single task learning and multitask learning approaches.
  arXiv  Detail & Related papers  (2023-05-10T14:20:34Z)
• Optimal transfer protocol by incremental layer defrosting [66.76153955485584]
  Transfer learning is a powerful tool enabling model training with limited amounts of data. The simplest transfer learning protocol is based on freezing" the feature-extractor layers of a network pre-trained on a data-rich source task. We show that this protocol is often sub-optimal and the largest performance gain may be achieved when smaller portions of the pre-trained network are kept frozen.
  arXiv  Detail & Related papers  (2023-03-02T17:32:11Z)
• Towards Estimating Transferability using Hard Subsets [25.86053764521497]
  We propose HASTE, a new strategy to estimate the transferability of a source model to a particular target task using only a harder subset of target data. We show that HASTE can be used with any existing transferability metric to improve their reliability. Our experimental results across multiple source model architectures, target datasets, and transfer learning tasks show that HASTE modified metrics are consistently better or on par with the state of the art transferability metrics.
  arXiv  Detail & Related papers  (2023-01-17T14:50:18Z)
• Active Multi-Task Representation Learning [50.13453053304159]
  We give the first formal study on resource task sampling by leveraging the techniques from active learning. We propose an algorithm that iteratively estimates the relevance of each source task to the target task and samples from each source task based on the estimated relevance.
  arXiv  Detail & Related papers  (2022-02-02T08:23:24Z)
• Unsupervised Multi-source Domain Adaptation Without Access to Source Data [58.551861130011886]
  Unsupervised Domain Adaptation (UDA) aims to learn a predictor model for an unlabeled domain by transferring knowledge from a separate labeled source domain. We propose a novel and efficient algorithm which automatically combines the source models with suitable weights in such a way that it performs at least as good as the best source model.
  arXiv  Detail & Related papers  (2021-04-05T10:45:12Z)
• Learning Centric Wireless Resource Allocation for Edge Computing: Algorithm and Experiment [15.577056429740951]
  Edge intelligence is an emerging network architecture that integrates sensing, communication, computing components, and supports various machine learning applications. Existing methods ignore two important facts: 1) different models have heterogeneous demands on training data; 2) there is a mismatch between the simulated environment and the real-world environment. This paper proposes the learning centric wireless resource allocation scheme that maximizes the worst learning performance of multiple tasks.
  arXiv  Detail & Related papers  (2020-10-29T06:20:40Z)
• Towards Accurate Knowledge Transfer via Target-awareness Representation Disentanglement [56.40587594647692]
  We propose a novel transfer learning algorithm, introducing the idea of Target-awareness REpresentation Disentanglement (TRED) TRED disentangles the relevant knowledge with respect to the target task from the original source model and used as a regularizer during fine-tuning the target model. Experiments on various real world datasets show that our method stably improves the standard fine-tuning by more than 2% in average.
  arXiv  Detail & Related papers  (2020-10-16T17:45:08Z)
• Dif-MAML: Decentralized Multi-Agent Meta-Learning [54.39661018886268]
  We propose a cooperative multi-agent meta-learning algorithm, referred to as MAML or Dif-MAML. We show that the proposed strategy allows a collection of agents to attain agreement at a linear rate and to converge to a stationary point of the aggregate MAML. Simulation results illustrate the theoretical findings and the superior performance relative to the traditional non-cooperative setting.
  arXiv  Detail & Related papers  (2020-10-06T16:51:09Z)
• XMixup: Efficient Transfer Learning with Auxiliary Samples by Cross-domain Mixup [60.07531696857743]
  Cross-domain Mixup (XMixup) improves the multitask paradigm for deep transfer learning. XMixup selects the auxiliary samples from the source dataset and augments training samples via the simple mixup strategy. Experiment results show that XMixup improves the accuracy by 1.9% on average.
  arXiv  Detail & Related papers  (2020-07-20T16:42:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.