Top-Tuning: a study on transfer learning for an efficient alternative to fine tuning for image classification with fast kernel methods

• URL: http://arxiv.org/abs/2209.07932v3
• Date: Thu, 9 Nov 2023 10:48:36 GMT
• Title: Top-Tuning: a study on transfer learning for an efficient alternative to fine tuning for image classification with fast kernel methods
• Authors: Paolo Didier Alfano, Vito Paolo Pastore, Lorenzo Rosasco, Francesca Odone
• Abstract summary: In this paper, we consider a simple transfer learning approach exploiting pre-trained convolutional features as input for a fast-to-train kernel method. We show that the top-tuning approach provides comparable accuracy with respect to fine-tuning, with a training time between one and two orders of magnitude smaller.
• Score: 12.325059377851485
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The impressive performance of deep learning architectures is associated with a massive increase in model complexity. Millions of parameters need to be tuned, with training and inference time scaling accordingly, together with energy consumption. But is massive fine-tuning always necessary? In this paper, focusing on image classification, we consider a simple transfer learning approach exploiting pre-trained convolutional features as input for a fast-to-train kernel method. We refer to this approach as \textit{top-tuning} since only the kernel classifier is trained on the target dataset. In our study, we perform more than 3000 training processes focusing on 32 small to medium-sized target datasets, a typical situation where transfer learning is necessary. We show that the top-tuning approach provides comparable accuracy with respect to fine-tuning, with a training time between one and two orders of magnitude smaller. These results suggest that top-tuning is an effective alternative to fine-tuning in small/medium datasets, being especially useful when training time efficiency and computational resources saving are crucial.

Related papers

• Center-Sensitive Kernel Optimization for Efficient On-Device Incremental Learning [88.78080749909665]
  Current on-device training methods just focus on efficient training without considering the catastrophic forgetting. This paper proposes a simple but effective edge-friendly incremental learning framework. Our method achieves average accuracy boost of 38.08% with even less memory and approximate computation.
  arXiv  Detail & Related papers  (2024-06-13T05:49:29Z)
• Time-, Memory- and Parameter-Efficient Visual Adaptation [75.28557015773217]
  We propose an adaptation method which does not backpropagate gradients through the backbone. We achieve this by designing a lightweight network in parallel that operates on features from the frozen, pretrained backbone.
  arXiv  Detail & Related papers  (2024-02-05T10:55:47Z)
• Improved Distribution Matching for Dataset Condensation [91.55972945798531]
  We propose a novel dataset condensation method based on distribution matching. Our simple yet effective method outperforms most previous optimization-oriented methods with much fewer computational resources.
  arXiv  Detail & Related papers  (2023-07-19T04:07:33Z)
• Adaptive Cross Batch Normalization for Metric Learning [75.91093210956116]
  Metric learning is a fundamental problem in computer vision. We show that it is equally important to ensure that the accumulated embeddings are up to date. In particular, it is necessary to circumvent the representational drift between the accumulated embeddings and the feature embeddings at the current training iteration.
  arXiv  Detail & Related papers  (2023-03-30T03:22:52Z)
• Less is More: Selective Layer Finetuning with SubTuning [26.43027780266698]
  Finetuning a pretrained model has become a standard approach for training neural networks on novel tasks, resulting in fast convergence and improved performance. In this work, we study an alternative finetuning method, where instead of finetuning all the weights of the network, we only train a carefully chosen subset of layers, keeping the rest of the weights frozen at their initial (pretrained) values. We demonstrate that emphsubset finetuning (or SubTuning) often achieves accuracy comparable to full finetuning of the model, and even surpasses the performance of full finetuning when training data is scarce.
  arXiv  Detail & Related papers  (2023-02-13T13:38:46Z)
• APE: Aligning Pretrained Encoders to Quickly Learn Aligned Multimodal Representations [21.651198216809608]
  We take advantage of existing unimodal encoders and careful curation of alignment data relevant to the downstream task of interest. We study a natural approach to aligning existing encoders via small auxiliary functions, and we find that this method is competitive with (or outperforms) state of the art in many settings.
  arXiv  Detail & Related papers  (2022-10-08T05:38:46Z)
• Beyond Transfer Learning: Co-finetuning for Action Localisation [64.07196901012153]
  We propose co-finetuning -- simultaneously training a single model on multiple upstream'' and downstream'' tasks. We demonstrate that co-finetuning outperforms traditional transfer learning when using the same total amount of data. We also show how we can easily extend our approach to multiple upstream'' datasets to further improve performance.
  arXiv  Detail & Related papers  (2022-07-08T10:25:47Z)
• FiT: Parameter Efficient Few-shot Transfer Learning for Personalized and Federated Image Classification [47.24770508263431]
  We develop FiLM Transfer (FiT) which fulfills requirements in the image classification setting. FiT uses an automatically configured Naive Bayes classifier on top of a fixed backbone that has been pretrained on large image datasets. We show that FiT achieves better classification accuracy than the state-of-the-art Big Transfer (BiT) algorithm at low-shot and on the challenging VTAB-1k benchmark.
  arXiv  Detail & Related papers  (2022-06-17T10:17:20Z)
• Adaptive Fine-Tuning of Transformer-Based Language Models for Named Entity Recognition [0.0]
  The current standard approach for fine-tuning language models includes a fixed number of training epochs and a linear learning rate schedule. In this paper, we introduce adaptive fine-tuning, which is an alternative approach that uses early stopping and a custom learning rate schedule.
  arXiv  Detail & Related papers  (2022-02-05T19:20:03Z)
• Fast Few-Shot Classification by Few-Iteration Meta-Learning [173.32497326674775]
  We introduce a fast optimization-based meta-learning method for few-shot classification. Our strategy enables important aspects of the base learner objective to be learned during meta-training. We perform a comprehensive experimental analysis, demonstrating the speed and effectiveness of our approach.
  arXiv  Detail & Related papers  (2020-10-01T15:59:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.