Characterization of Transfer Using Multi-task Learning Curves

• URL: http://arxiv.org/abs/2512.24866v1
• Date: Wed, 31 Dec 2025 13:55:18 GMT
• Title: Characterization of Transfer Using Multi-task Learning Curves
• Authors: András Millinghoffer, Bence Bolgár, Péter Antal,
• Abstract summary: We show that perturbing the data set by including more samples, instead of perturbing the model by updates gradient, provides a complementary and more fundamental characterization of transfer effects.<n>We model transfer effects using multi-task learning curves approximating the inductive performance over varying sample sizes.<n>Our results show that learning curves can better capture the effects of multi-task learning and their multi-task extensions can delineate pairwise and contextual transfer effects in foundation models.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Transfer effects manifest themselves both during training using a fixed data set and in inductive inference using accumulating data. We hypothesize that perturbing the data set by including more samples, instead of perturbing the model by gradient updates, provides a complementary and more fundamental characterization of transfer effects. To capture this phenomenon, we quantitatively model transfer effects using multi-task learning curves approximating the inductive performance over varying sample sizes. We describe an efficient method to approximate multi-task learning curves analogous to the Task Affinity Grouping method applied during training. We compare the statistical and computational approaches to transfer, which indicates considerably higher compute costs for the previous but better power and broader applicability. Evaluations are performed using a benchmark drug-target interaction data set. Our results show that learning curves can better capture the effects of multi-task learning and their multi-task extensions can delineate pairwise and contextual transfer effects in foundation models.

Related papers

• Latent Traits and Cross-Task Transfer: Deconstructing Dataset Interactions in LLM Fine-tuning [8.082936847467638]
  We propose an analysis framework, building a transfer learning matrix and dimensionality reduction, to dissect cross-task interactions.<n>We train and analyze 10 models to identify latent abilities (e.g., Reasoning, Sentiment Classification, NLU, Arithmetic) and discover the side effects of the transfer learning.
  arXiv  Detail & Related papers  (2025-09-17T01:45:42Z)
• Understanding Data Influence with Differential Approximation [63.817689230826595]
  We introduce a new formulation to approximate a sample's influence by accumulating the differences in influence between consecutive learning steps, which we term Diff-In.<n>By employing second-order approximations, we approximate these difference terms with high accuracy while eliminating the need for model convexity required by existing methods.<n>Our theoretical analysis demonstrates that Diff-In achieves significantly lower approximation error compared to existing influence estimators.
  arXiv  Detail & Related papers  (2025-08-20T11:59:32Z)
• Exploring Transfer Learning for Deep Learning Polyp Detection in Colonoscopy Images Using YOLOv8 [4.596575711979469]
  Transfer learning techniques leverage knowledge from pre-training on related datasets.<n>Finding the right dataset for pre-training can play a critical role in determining the success of transfer learning.<n>We show that models pre-trained on relevant datasets consistently outperform those trained from scratch.
  arXiv  Detail & Related papers  (2025-01-31T19:33:45Z)
• Capturing the Temporal Dependence of Training Data Influence [100.91355498124527]
  We formalize the concept of trajectory-specific leave-one-out influence, which quantifies the impact of removing a data point during training.<n>We propose data value embedding, a novel technique enabling efficient approximation of trajectory-specific LOO.<n>As data value embedding captures training data ordering, it offers valuable insights into model training dynamics.
  arXiv  Detail & Related papers  (2024-12-12T18:28:55Z)
• Dissecting Representation Misalignment in Contrastive Learning via Influence Function [15.28417468377201]
  We introduce the Extended Influence Function for Contrastive Loss (ECIF), an influence function crafted for contrastive loss.<n>ECIF considers both positive and negative samples and provides a closed-form approximation of contrastive learning models.<n>Building upon ECIF, we develop a series of algorithms for data evaluation, misalignment detection, and misprediction trace-back tasks.
  arXiv  Detail & Related papers  (2024-11-18T15:45:41Z)
• On the Trade-off of Intra-/Inter-class Diversity for Supervised Pre-training [72.8087629914444]
  We study the impact of the trade-off between the intra-class diversity (the number of samples per class) and the inter-class diversity (the number of classes) of a supervised pre-training dataset. With the size of the pre-training dataset fixed, the best downstream performance comes with a balance on the intra-/inter-class diversity.
  arXiv  Detail & Related papers  (2023-05-20T16:23:50Z)
• An Exploration of Data Efficiency in Intra-Dataset Task Transfer for Dialog Understanding [65.75873687351553]
  This study explores the effects of varying quantities of target task training data on sequential transfer learning in the dialog domain. Unintuitively, our data shows that often target task training data size has minimal effect on how sequential transfer learning performs compared to the same model without transfer learning.
  arXiv  Detail & Related papers  (2022-10-21T04:36:46Z)
• Intra-domain and cross-domain transfer learning for time series data -- How transferable are the features? [0.0]
  This study aims to assess how transferable are the features between different domains of time series data. The effects of transfer learning are observed in terms of predictive performance of the models and their convergence rate during training.
  arXiv  Detail & Related papers  (2022-01-12T12:55:21Z)
• Multi-Stage Influence Function [97.19210942277354]
  We develop a multi-stage influence function score to track predictions from a finetuned model all the way back to the pretraining data. We study two different scenarios with the pretrained embeddings fixed or updated in the finetuning tasks.
  arXiv  Detail & Related papers  (2020-07-17T16:03:11Z)
• Task-Feature Collaborative Learning with Application to Personalized Attribute Prediction [166.87111665908333]
  We propose a novel multi-task learning method called Task-Feature Collaborative Learning (TFCL) Specifically, we first propose a base model with a heterogeneous block-diagonal structure regularizer to leverage the collaborative grouping of features and tasks. As a practical extension, we extend the base model by allowing overlapping features and differentiating the hard tasks.
  arXiv  Detail & Related papers  (2020-04-29T02:32:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.