Fast and Accurate Transferability Measurement by Evaluating Intra-class Feature Variance

• URL: http://arxiv.org/abs/2308.05986v1
• Date: Fri, 11 Aug 2023 07:50:40 GMT
• Title: Fast and Accurate Transferability Measurement by Evaluating Intra-class Feature Variance
• Authors: Huiwen Xu, U Kang
• Abstract summary: Transferability measurement is to quantify how transferable is a pre-trained model learned on a source task to a target task. We propose TMI (TRANSFERABILITY MEASUREMENT WITH INTRA-CLASS FEATURE VARIANCE), a fast and accurate algorithm to measure transferability.
• Score: 20.732095457775138
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Given a set of pre-trained models, how can we quickly and accurately find the most useful pre-trained model for a downstream task? Transferability measurement is to quantify how transferable is a pre-trained model learned on a source task to a target task. It is used for quickly ranking pre-trained models for a given task and thus becomes a crucial step for transfer learning. Existing methods measure transferability as the discrimination ability of a source model for a target data before transfer learning, which cannot accurately estimate the fine-tuning performance. Some of them restrict the application of transferability measurement in selecting the best supervised pre-trained models that have classifiers. It is important to have a general method for measuring transferability that can be applied in a variety of situations, such as selecting the best self-supervised pre-trained models that do not have classifiers, and selecting the best transferring layer for a target task. In this work, we propose TMI (TRANSFERABILITY MEASUREMENT WITH INTRA-CLASS FEATURE VARIANCE), a fast and accurate algorithm to measure transferability. We view transferability as the generalization of a pre-trained model on a target task by measuring intra-class feature variance. Intra-class variance evaluates the adaptability of the model to a new task, which measures how transferable the model is. Compared to previous studies that estimate how discriminative the models are, intra-class variance is more accurate than those as it does not require an optimal feature extractor and classifier. Extensive experiments on real-world datasets show that TMI outperforms competitors for selecting the top-5 best models, and exhibits consistently better correlation in 13 out of 17 cases.

Related papers

• Efficient Transferability Assessment for Selection of Pre-trained Detectors [63.21514888618542]
  This paper studies the efficient transferability assessment of pre-trained object detectors. We build up a detector transferability benchmark which contains a large and diverse zoo of pre-trained detectors. Experimental results demonstrate that our method outperforms other state-of-the-art approaches in assessing transferability.
  arXiv  Detail & Related papers  (2024-03-14T14:23:23Z)
• Building a Winning Team: Selecting Source Model Ensembles using a Submodular Transferability Estimation Approach [20.86345962679122]
  Estimating the transferability of publicly available pretrained models to a target task has assumed an important place for transfer learning tasks. We propose a novel Optimal tranSport-based suBmOdular tRaNsferability metric (OSBORN) to estimate the transferability of an ensemble of models to a downstream task.
  arXiv  Detail & Related papers  (2023-09-05T17:57:31Z)
• Towards Estimating Transferability using Hard Subsets [25.86053764521497]
  We propose HASTE, a new strategy to estimate the transferability of a source model to a particular target task using only a harder subset of target data. We show that HASTE can be used with any existing transferability metric to improve their reliability. Our experimental results across multiple source model architectures, target datasets, and transfer learning tasks show that HASTE modified metrics are consistently better or on par with the state of the art transferability metrics.
  arXiv  Detail & Related papers  (2023-01-17T14:50:18Z)
• Self-Distillation for Further Pre-training of Transformers [83.84227016847096]
  We propose self-distillation as a regularization for a further pre-training stage. We empirically validate the efficacy of self-distillation on a variety of benchmark datasets for image and text classification tasks.
  arXiv  Detail & Related papers  (2022-09-30T02:25:12Z)
• SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilities [76.97949110580703]
  We introduce SUPERB-SG, a new benchmark to evaluate pre-trained models across various speech tasks. We use a lightweight methodology to test the robustness of representations learned by pre-trained models under shifts in data domain. We also show that the task diversity of SUPERB-SG coupled with limited task supervision is an effective recipe for evaluating the generalizability of model representation.
  arXiv  Detail & Related papers  (2022-03-14T04:26:40Z)
• Model-Agnostic Multitask Fine-tuning for Few-shot Vision-Language Transfer Learning [59.38343286807997]
  We propose Model-Agnostic Multitask Fine-tuning (MAMF) for vision-language models on unseen tasks. Compared with model-agnostic meta-learning (MAML), MAMF discards the bi-level optimization and uses only first-order gradients. We show that MAMF consistently outperforms the classical fine-tuning method for few-shot transfer learning on five benchmark datasets.
  arXiv  Detail & Related papers  (2022-03-09T17:26:53Z)
• Transferability Estimation using Bhattacharyya Class Separability [37.52588126267552]
  Transfer learning is a popular method for leveraging pre-trained models in computer vision. It is difficult to quantify which pre-trained source models are suitable for a specific target task. We propose a novel method for quantifying transferability between a source model and a target dataset.
  arXiv  Detail & Related papers  (2021-11-24T20:22:28Z)
• TransTailor: Pruning the Pre-trained Model for Improved Transfer Learning [5.9292619981667976]
  We propose TransTailor, targeting at pruning the pre-trained model for improved transfer learning. We prune and fine-tune the pre-trained model according to the target-aware weight importance. We transfer a more suitable sub-structure that can be applied during fine-tuning to benefit the final performance.
  arXiv  Detail & Related papers  (2021-03-02T07:58:35Z)
• LogME: Practical Assessment of Pre-trained Models for Transfer Learning [80.24059713295165]
  The Logarithm of Maximum Evidence (LogME) can be used to assess pre-trained models for transfer learning. Compared to brute-force fine-tuning, LogME brings over $3000times$ speedup in wall-clock time.
  arXiv  Detail & Related papers  (2021-02-22T13:58:11Z)
• Do Adversarially Robust ImageNet Models Transfer Better? [102.09335596483695]
  adversarially robust models often perform better than their standard-trained counterparts when used for transfer learning. Our results are consistent with (and in fact, add to) recent hypotheses stating that robustness leads to improved feature representations.
  arXiv  Detail & Related papers  (2020-07-16T17:42:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.