Practical Transferability Estimation for Image Classification Tasks
- URL: http://arxiv.org/abs/2106.10479v3
- Date: Thu, 29 Feb 2024 06:29:09 GMT
- Title: Practical Transferability Estimation for Image Classification Tasks
- Authors: Yang Tan, Yang Li, Shao-Lun Huang
- Abstract summary: A major challenge is how to make transfereability estimation robust under the cross-domain cross-task settings.
The recently proposed OTCE score solves this problem by considering both domain and task differences.
We propose a practical transferability metric called JC-NCE score that dramatically improves the robustness of the task difference estimation.
- Score: 20.07223947190349
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Transferability estimation is an essential problem in transfer learning to
predict how good the performance is when transferring a source model (or source
task) to a target task. Recent analytical transferability metrics have been
widely used for source model selection and multi-task learning. A major
challenge is how to make transfereability estimation robust under the
cross-domain cross-task settings. The recently proposed OTCE score solves this
problem by considering both domain and task differences, with the help of
transfer experiences on auxiliary tasks, which causes an efficiency overhead.
In this work, we propose a practical transferability metric called JC-NCE score
that dramatically improves the robustness of the task difference estimation in
OTCE, thus removing the need for auxiliary tasks. Specifically, we build the
joint correspondences between source and target data via solving an optimal
transport problem with a ground cost considering both the sample distance and
label distance, and then compute the transferability score as the negative
conditional entropy of the matched labels. Extensive validations under the
intra-dataset and inter-dataset transfer settings demonstrate that our JC-NCE
score outperforms the auxiliary-task free version of OTCE for 7% and 12%,
respectively, and is also more robust than other existing transferability
metrics on average.
Related papers
- ForkMerge: Mitigating Negative Transfer in Auxiliary-Task Learning [59.08197876733052]
Auxiliary-Task Learning (ATL) aims to improve the performance of the target task by leveraging the knowledge obtained from related tasks.
Sometimes, learning multiple tasks simultaneously results in lower accuracy than learning only the target task, known as negative transfer.
ForkMerge is a novel approach that periodically forks the model into multiple branches, automatically searches the varying task weights.
arXiv Detail & Related papers (2023-01-30T02:27:02Z) - Towards Estimating Transferability using Hard Subsets [25.86053764521497]
We propose HASTE, a new strategy to estimate the transferability of a source model to a particular target task using only a harder subset of target data.
We show that HASTE can be used with any existing transferability metric to improve their reliability.
Our experimental results across multiple source model architectures, target datasets, and transfer learning tasks show that HASTE modified metrics are consistently better or on par with the state of the art transferability metrics.
arXiv Detail & Related papers (2023-01-17T14:50:18Z) - Transferability Estimation Based On Principal Gradient Expectation [68.97403769157117]
Cross-task transferability is compatible with transferred results while keeping self-consistency.
Existing transferability metrics are estimated on the particular model by conversing source and target tasks.
We propose Principal Gradient Expectation (PGE), a simple yet effective method for assessing transferability across tasks.
arXiv Detail & Related papers (2022-11-29T15:33:02Z) - Transferability-Guided Cross-Domain Cross-Task Transfer Learning [21.812715282796255]
We propose two novel transferability metrics F-OTCE and JC-OTCE.
F-OTCE estimates transferability by first solving an Optimal Transport problem between source and target distributions.
JC-OTCE improves the transferability of F-OTCE by including label distances in the OT problem.
arXiv Detail & Related papers (2022-07-12T13:06:16Z) - Identifying Suitable Tasks for Inductive Transfer Through the Analysis
of Feature Attributions [78.55044112903148]
We use explainability techniques to predict whether task pairs will be complementary, through comparison of neural network activation between single-task models.
Our results show that, through this approach, it is possible to reduce training time by up to 83.5% at a cost of only 0.034 reduction in positive-class F1 on the TREC-IS 2020-A dataset.
arXiv Detail & Related papers (2022-02-02T15:51:07Z) - Transferability Estimation for Semantic Segmentation Task [20.07223947190349]
We extend the recent transferability metric OTCE score to the semantic segmentation task.
The challenge in applying the OTCE score is the high dimensional segmentation output, which is difficult to find the optimal coupling between so many pixels under an acceptable cost.
Experimental evaluation on Cityscapes, BDD100K and GTA5 datasets demonstrates that the OTCE score highly correlates with the transfer performance.
arXiv Detail & Related papers (2021-09-30T16:21:17Z) - Frustratingly Easy Transferability Estimation [64.42879325144439]
We propose a simple, efficient, and effective transferability measure named TransRate.
TransRate measures the transferability as the mutual information between the features of target examples extracted by a pre-trained model and labels of them.
Despite its extraordinary simplicity in 10 lines of codes, TransRate performs remarkably well in extensive evaluations on 22 pre-trained models and 16 downstream tasks.
arXiv Detail & Related papers (2021-06-17T10:27:52Z) - OTCE: A Transferability Metric for Cross-Domain Cross-Task
Representations [6.730043708859326]
We propose a transferability metric called Optimal Transport based Conditional Entropy (OTCE)
OTCE characterizes transferability as a combination of domain difference and task difference, and explicitly evaluates them from data in a unified framework.
Experiments on the largest cross-domain dataset DomainNet and Office31 demonstrate that OTCE shows an average of 21% gain in the correlation with the ground truth transfer accuracy.
arXiv Detail & Related papers (2021-03-25T13:51:33Z) - Learning Invariant Representations across Domains and Tasks [81.30046935430791]
We propose a novel Task Adaptation Network (TAN) to solve this unsupervised task transfer problem.
In addition to learning transferable features via domain-adversarial training, we propose a novel task semantic adaptor that uses the learning-to-learn strategy to adapt the task semantics.
TAN significantly increases the recall and F1 score by 5.0% and 7.8% compared to recently strong baselines.
arXiv Detail & Related papers (2021-03-03T11:18:43Z) - Exploring and Predicting Transferability across NLP Tasks [115.6278033699853]
We study the transferability between 33 NLP tasks across three broad classes of problems.
Our results show that transfer learning is more beneficial than previously thought.
We also develop task embeddings that can be used to predict the most transferable source tasks for a given target task.
arXiv Detail & Related papers (2020-05-02T09:39:36Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.