Boosting Memory Efficiency in Transfer Learning for High-Resolution Medical Image Classification

• URL: http://arxiv.org/abs/2408.02426v2
• Date: Thu, 02 Jan 2025 15:25:43 GMT
• Title: Boosting Memory Efficiency in Transfer Learning for High-Resolution Medical Image Classification
• Authors: Yijin Huang, Pujin Cheng, Roger Tam, Xiaoying Tang,
• Abstract summary: Fine-grained Prompt Tuning plus (FPT+) is a PETL method designed for high-resolution medical image classification. FPT+ performs transfer learning by training a lightweight side network and accessing pre-trained knowledge from a large pre-trained model. We evaluate FPT+ on eight medical image datasets of varying sizes, modalities, and complexities.
• Score: 1.5791081894226173
• License:
• Abstract: The success of large-scale pre-trained models has established fine-tuning as a standard method for achieving significant improvements in downstream tasks. However, fine-tuning the entire parameter set of a pre-trained model is costly. Parameter-efficient transfer learning (PETL) has recently emerged as a cost-effective alternative for adapting pre-trained models to downstream tasks. Despite its advantages, the increasing model size and input resolution present challenges for PETL, as the training memory consumption is not reduced as effectively as the parameter usage. In this paper, we introduce Fine-grained Prompt Tuning plus (FPT+), a PETL method designed for high-resolution medical image classification, which significantly reduces the training memory consumption compared to other PETL methods. FPT+ performs transfer learning by training a lightweight side network and accessing pre-trained knowledge from a large pre-trained model (LPM) through fine-grained prompts and fusion modules. Specifically, we freeze the LPM of interest and construct a learnable lightweight side network. The frozen LPM processes high-resolution images to extract fine-grained features, while the side network employs corresponding down-sampled low-resolution images to minimize the memory usage. To enable the side network to leverage pre-trained knowledge, we propose fine-grained prompts and fusion modules, which collaborate to summarize information through the LPM's intermediate activations. We evaluate FPT+ on eight medical image datasets of varying sizes, modalities, and complexities. Experimental results demonstrate that FPT+ outperforms other PETL methods, using only 1.03% of the learnable parameters and 3.18% of the memory required for fine-tuning an entire ViT-B model. Our code is available https://github.com/YijinHuang/FPT.

Related papers

• SHERL: Synthesizing High Accuracy and Efficient Memory for Resource-Limited Transfer Learning [63.93193829913252]
  We propose an innovative METL strategy called SHERL for resource-limited scenarios. In the early route, intermediate outputs are consolidated via an anti-redundancy operation. In the late route, utilizing minimal late pre-trained layers could alleviate the peak demand on memory overhead.
  arXiv  Detail & Related papers  (2024-07-10T10:22:35Z)
• Fine-grained Prompt Tuning: A Parameter and Memory Efficient Transfer Learning Method for High-resolution Medical Image Classification [1.5791081894226173]
  Fine-grained Prompt Tuning (FPT) is a cost-effective way to transfer pre-trained models to downstream tasks. FPT significantly reduces memory consumption compared to other PETL methods. We evaluate FPT on four medical datasets with varying sizes, modalities, and complexities.
  arXiv  Detail & Related papers  (2024-03-12T12:05:43Z)
• Time-, Memory- and Parameter-Efficient Visual Adaptation [75.28557015773217]
  We propose an adaptation method which does not backpropagate gradients through the backbone. We achieve this by designing a lightweight network in parallel that operates on features from the frozen, pretrained backbone.
  arXiv  Detail & Related papers  (2024-02-05T10:55:47Z)
• DTL: Disentangled Transfer Learning for Visual Recognition [21.549234013998255]
  We introduce Disentangled Transfer Learning (DTL), which disentangles the trainable parameters from the backbone using a lightweight Compact Side Network (CSN) The proposed method not only reduces a large amount of GPU memory usage and trainable parameters, but also outperforms existing PETL methods by a significant margin in accuracy.
  arXiv  Detail & Related papers  (2023-12-13T02:51:26Z)
• Parameter and Computation Efficient Transfer Learning for Vision-Language Pre-trained Models [79.34513906324727]
  In this paper, we aim at parameter and efficient transfer learning (PCETL) for vision-language pre-trained models. We propose a novel dynamic architecture skipping (DAS) approach towards effective PCETL.
  arXiv  Detail & Related papers  (2023-09-04T09:34:33Z)
• UniPT: Universal Parallel Tuning for Transfer Learning with Efficient Parameter and Memory [69.33445217944029]
  PETL is an effective strategy for adapting pre-trained models to downstream domains. Recent PETL works focus on the more valuable memory-efficient characteristic. We propose a new memory-efficient PETL strategy, Universal Parallel Tuning (UniPT)
  arXiv  Detail & Related papers  (2023-08-28T05:38:43Z)
• Approximated Prompt Tuning for Vision-Language Pre-trained Models [54.326232586461614]
  In vision-language pre-trained models, prompt tuning often requires a large number of learnable tokens to bridge the gap between the pre-training and downstream tasks. We propose a novel Approximated Prompt Tuning (APT) approach towards efficient VL transfer learning.
  arXiv  Detail & Related papers  (2023-06-27T05:43:47Z)
• Make Pre-trained Model Reversible: From Parameter to Memory Efficient Fine-Tuning [6.451743797015637]
  We propose memory-efficient fine-tuning (MEFT) for pre-trained language models. MEFT inserts adapters into a PLM, preserving the PLM's starting point and making it reversible without additional pre-training. MEFT significantly reduces the activation memory up to 84% of full fine-tuning with a negligible amount of trainable parameters.
  arXiv  Detail & Related papers  (2023-06-01T09:26:17Z)
• Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning [81.3514358542452]
  Few-shot in-context learning (ICL) incurs substantial computational, memory, and storage costs because it involves processing all of the training examples every time a prediction is made. parameter-efficient fine-tuning offers an alternative paradigm where a small set of parameters are trained to enable a model to perform the new task. In this paper, we rigorously compare few-shot ICL and parameter-efficient fine-tuning and demonstrate that the latter offers better accuracy as well as dramatically lower computational costs.
  arXiv  Detail & Related papers  (2022-05-11T17:10:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.