To transfer or not transfer: Unified transferability metric and analysis

• URL: http://arxiv.org/abs/2305.07741v1
• Date: Fri, 12 May 2023 19:52:11 GMT
• Title: To transfer or not transfer: Unified transferability metric and analysis
• Abstract summary: In transfer learning, transferability is one of the most fundamental problems, which aims to evaluate the effectiveness of arbitrary transfer tasks. We propose a new analytical approach and metric, Wasserstein Distance based Joint Estimation, for transferability estimation. We approximate the target transfer risk by proposing a non-symmetric, easy-to-understand and easy-to-calculate target risk bound.
• Score: 13.274835852615576
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In transfer learning, transferability is one of the most fundamental problems, which aims to evaluate the effectiveness of arbitrary transfer tasks. Existing research focuses on classification tasks and neglects domain or task differences. More importantly, there is a lack of research to determine whether to transfer or not. To address these, we propose a new analytical approach and metric, Wasserstein Distance based Joint Estimation (WDJE), for transferability estimation and determination in a unified setting: classification and regression problems with domain and task differences. The WDJE facilitates decision-making on whether to transfer or not by comparing the target risk with and without transfer. To enable the comparison, we approximate the target transfer risk by proposing a non-symmetric, easy-to-understand and easy-to-calculate target risk bound that is workable even with limited target labels. The proposed bound relates the target risk to source model performance, domain and task differences based on Wasserstein distance. We also extend our bound into unsupervised settings and establish the generalization bound from finite empirical samples. Our experiments in image classification and remaining useful life regression prediction illustrate the effectiveness of the WDJE in determining whether to transfer or not, and the proposed bound in approximating the target transfer risk.

Related papers

• Cross-Domain Data Selection and Augmentation for Automatic Compliance Detection [1.8314346918344617]
  We study data selection as a strategy to mitigate negative transfer in compliance detection framed as a natural language inference (NLI) task.<n>Specifically, we evaluate four approaches for selecting augmentation from a larger source domain.<n>Our findings demonstrate that targeted data selection substantially reduces negative transfer, offering a practical path toward scalable and reliable compliance automation across heterogeneous regulations.
  arXiv  Detail & Related papers  (2026-04-23T09:28:50Z)
• Transfer Learning Through Conditional Quantile Matching [3.86972243789112]
  We introduce a transfer learning framework for regression that leverages heterogeneous source domains to improve predictive performance in a data-scarce target domain.<n>Our approach learns a conditional generative model separately for each source domain and calibrates the generated responses to the target domain via conditional quantile matching.
  arXiv  Detail & Related papers  (2026-02-02T17:19:55Z)
• Heterogeneous transfer learning for high dimensional regression with feature mismatch [1.6385815610837167]
  We consider the problem of transferring knowledge from a source, or proxy, domain to a new target domain for learning a high-dimensional regression model with possibly different features. Most homogeneous transfer and multi-task learning methods assume that the target and proxy domains have the same feature space. We propose a two-stage method that involves learning the relationship between the missing and observed features through a projection step in the proxy data.
  arXiv  Detail & Related papers  (2024-12-24T01:29:31Z)
• Robust Transfer Learning with Unreliable Source Data [13.276850367115333]
  We introduce a novel quantity called the ''ambiguity level'' that measures the discrepancy between the target and source regression functions. We propose a simple transfer learning procedure, and establish a general theorem that shows how this new quantity is related to the transferability of learning.
  arXiv  Detail & Related papers  (2023-10-06T21:50:21Z)
• Logit Margin Matters: Improving Transferable Targeted Adversarial Attack by Logit Calibration [85.71545080119026]
  Cross-Entropy (CE) loss function is insufficient to learn transferable targeted adversarial examples. We propose two simple and effective logit calibration methods, which are achieved by downscaling the logits with a temperature factor and an adaptive margin. Experiments conducted on the ImageNet dataset validate the effectiveness of the proposed methods.
  arXiv  Detail & Related papers  (2023-03-07T06:42:52Z)
• Transferability Estimation Based On Principal Gradient Expectation [68.97403769157117]
  Cross-task transferability is compatible with transferred results while keeping self-consistency. Existing transferability metrics are estimated on the particular model by conversing source and target tasks. We propose Principal Gradient Expectation (PGE), a simple yet effective method for assessing transferability across tasks.
  arXiv  Detail & Related papers  (2022-11-29T15:33:02Z)
• Estimation and inference for transfer learning with high-dimensional quantile regression [3.4510296013600374]
  We propose a transfer learning procedure in the framework of high-dimensional quantile regression models. We establish error bounds of transfer learning estimator based on delicately selected transferable source domains. By adopting data-splitting technique, we advocate a transferability detection approach that guarantees to circumvent negative transfer.
  arXiv  Detail & Related papers  (2022-11-26T14:40:19Z)
• Learning Unbiased Transferability for Domain Adaptation by Uncertainty Modeling [107.24387363079629]
  Domain adaptation aims to transfer knowledge from a labeled source domain to an unlabeled or a less labeled but related target domain. Due to the imbalance between the amount of annotated data in the source and target domains, only the target distribution is aligned to the source domain. We propose a non-intrusive Unbiased Transferability Estimation Plug-in (UTEP) by modeling the uncertainty of a discriminator in adversarial-based DA methods to optimize unbiased transfer.
  arXiv  Detail & Related papers  (2022-06-02T21:58:54Z)
• Practical Transferability Estimation for Image Classification Tasks [20.07223947190349]
  A major challenge is how to make transfereability estimation robust under the cross-domain cross-task settings. The recently proposed OTCE score solves this problem by considering both domain and task differences. We propose a practical transferability metric called JC-NCE score that dramatically improves the robustness of the task difference estimation.
  arXiv  Detail & Related papers  (2021-06-19T11:59:11Z)
• Frustratingly Easy Transferability Estimation [64.42879325144439]
  We propose a simple, efficient, and effective transferability measure named TransRate. TransRate measures the transferability as the mutual information between the features of target examples extracted by a pre-trained model and labels of them. Despite its extraordinary simplicity in 10 lines of codes, TransRate performs remarkably well in extensive evaluations on 22 pre-trained models and 16 downstream tasks.
  arXiv  Detail & Related papers  (2021-06-17T10:27:52Z)
• Towards Accurate Knowledge Transfer via Target-awareness Representation Disentanglement [56.40587594647692]
  We propose a novel transfer learning algorithm, introducing the idea of Target-awareness REpresentation Disentanglement (TRED) TRED disentangles the relevant knowledge with respect to the target task from the original source model and used as a regularizer during fine-tuning the target model. Experiments on various real world datasets show that our method stably improves the standard fine-tuning by more than 2% in average.
  arXiv  Detail & Related papers  (2020-10-16T17:45:08Z)
• Learning Calibrated Uncertainties for Domain Shift: A Distributionally Robust Learning Approach [150.8920602230832]
  We propose a framework for learning calibrated uncertainties under domain shifts. In particular, the density ratio estimation reflects the closeness of a target (test) sample to the source (training) distribution. We show that our proposed method generates calibrated uncertainties that benefit downstream tasks.
  arXiv  Detail & Related papers  (2020-10-08T02:10:54Z)
• CSCL: Critical Semantic-Consistent Learning for Unsupervised Domain Adaptation [42.226842513334184]
  We develop a new Critical Semantic-Consistent Learning model, which mitigates the discrepancy of both domain-wise and category-wise distributions. Specifically, a critical transfer based adversarial framework is designed to highlight transferable domain-wise knowledge while neglecting untransferable knowledge.
  arXiv  Detail & Related papers  (2020-08-24T14:12:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.