Probing the Efficacy of Federated Parameter-Efficient Fine-Tuning of Vision Transformers for Medical Image Classification

• URL: http://arxiv.org/abs/2407.11573v1
• Date: Tue, 16 Jul 2024 10:28:50 GMT
• Title: Probing the Efficacy of Federated Parameter-Efficient Fine-Tuning of Vision Transformers for Medical Image Classification
• Authors: Naif Alkhunaizi, Faris Almalik, Rouqaiah Al-Refai, Muzammal Naseer, Karthik Nandakumar,
• Abstract summary: Fine-tuning pre-trained models for various downstream tasks is a critical problem in the medical imaging domain. Large size of these models necessitates the use of parameter-efficient fine-tuning (PEFT) to reduce the communication burden in federated learning. In this work, we investigate various federated PEFT strategies for adapting a Vision Transformer (ViT) model for medical image classification.
• Score: 16.070261684997362
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: With the advent of large pre-trained transformer models, fine-tuning these models for various downstream tasks is a critical problem. Paucity of training data, the existence of data silos, and stringent privacy constraints exacerbate this fine-tuning problem in the medical imaging domain, creating a strong need for algorithms that enable collaborative fine-tuning of pre-trained models. Moreover, the large size of these models necessitates the use of parameter-efficient fine-tuning (PEFT) to reduce the communication burden in federated learning. In this work, we systematically investigate various federated PEFT strategies for adapting a Vision Transformer (ViT) model (pre-trained on a large natural image dataset) for medical image classification. Apart from evaluating known PEFT techniques, we introduce new federated variants of PEFT algorithms such as visual prompt tuning (VPT), low-rank decomposition of visual prompts, stochastic block attention fine-tuning, and hybrid PEFT methods like low-rank adaptation (LoRA)+VPT. Moreover, we perform a thorough empirical analysis to identify the optimal PEFT method for the federated setting and understand the impact of data distribution on federated PEFT, especially for out-of-domain (OOD) and non-IID data. The key insight of this study is that while most federated PEFT methods work well for in-domain transfer, there is a substantial accuracy vs. efficiency trade-off when dealing with OOD and non-IID scenarios, which is commonly the case in medical imaging. Specifically, every order of magnitude reduction in fine-tuned/exchanged parameters can lead to a 4% drop in accuracy. Thus, the initial model choice is crucial for federated PEFT. It is preferable to use medical foundation models learned from in-domain medical image data (if available) rather than general vision models.

Related papers

• Visual Fourier Prompt Tuning [63.66866445034855]
  We propose the Visual Fourier Prompt Tuning (VFPT) method as a general and effective solution for adapting large-scale transformer-based models. Our approach incorporates the Fast Fourier Transform into prompt embeddings and harmoniously considers both spatial and frequency domain information. Our results demonstrate that our approach outperforms current state-of-the-art baselines on two benchmarks.
  arXiv  Detail & Related papers  (2024-11-02T18:18:35Z)
• Embedded Prompt Tuning: Towards Enhanced Calibration of Pretrained Models for Medical Images [18.094731760514264]
  We study the effectiveness of fine-tuning methods when adapting foundation models to medical image classification tasks. We propose the Embedded Prompt Tuning (EPT) method by embedding prompt tokens into the expanded channels. EPT outperforms several state-of-the-art finetuning methods by a significant margin on few-shot medical image classification tasks.
  arXiv  Detail & Related papers  (2024-07-01T06:35:53Z)
• Communication-Efficient Hybrid Federated Learning for E-health with Horizontal and Vertical Data Partitioning [67.49221252724229]
  E-health allows smart devices and medical institutions to collaboratively collect patients' data, which is trained by Artificial Intelligence (AI) technologies to help doctors make diagnosis. Applying federated learning in e-health faces many challenges. Medical data is both horizontally and vertically partitioned. A naive combination of HFL and VFL has limitations including low training efficiency, unsound convergence analysis, and lack of parameter tuning strategies.
  arXiv  Detail & Related papers  (2024-04-15T19:45:07Z)
• MEDDAP: Medical Dataset Enhancement via Diversified Augmentation Pipeline [1.4910709350090976]
  We introduce a novel pipeline called MEDDAP to augment existing small datasets by automatically generating new informative labeled samples. USLoRA allows for selective fine-tuning of weights within SD, requiring fewer than 0.1% of parameters compared to fully fine-tuning only the UNet portion of SD. This approach is inspired by clinicians' decision-making processes regarding breast tumors, where tumor shape often plays a more crucial role than intensity.
  arXiv  Detail & Related papers  (2024-03-25T00:17:43Z)
• Less Could Be Better: Parameter-efficient Fine-tuning Advances Medical Vision Foundation Models [71.18275399694689]
  The effectiveness of PEFT on medical vision foundation models is still unclear. We set up new state-of-the-art on a range of data-efficient learning tasks, such as an AUROC score of 80.6% using 1% labeled data on NIH ChestX-ray14. We hope this study can evoke more attention from the community in the use of PEFT for transfer learning on medical imaging tasks.
  arXiv  Detail & Related papers  (2024-01-22T18:59:07Z)
• DVPT: Dynamic Visual Prompt Tuning of Large Pre-trained Models for Medical Image Analysis [30.608225734194416]
  We propose a dynamic visual prompt tuning method, named DVPT, for medical image analysis. It can extract knowledge beneficial to downstream tasks from large models with a few trainable parameters. It can save up to 60% labeled data and 99% storage cost of ViT-B/16.
  arXiv  Detail & Related papers  (2023-07-19T07:11:11Z)
• Parameter-Efficient Fine-Tuning for Medical Image Analysis: The Missed Opportunity [15.404013190033242]
  The application of. Efficient Fine-Tuning (PEFT) in medical image analysis is relatively unexplored. This study fills this gap by evaluating 17 distinct PEFT algorithms on image classification and text-to-image generation tasks. Our findings demonstrate PEFT's effectiveness, particularly in low data regimes common in medical imaging.
  arXiv  Detail & Related papers  (2023-05-14T21:18:18Z)
• Strong Baselines for Parameter Efficient Few-Shot Fine-tuning [50.83426196335385]
  Few-shot classification (FSC) entails learning novel classes given only a few examples per class after a pre-training (or meta-training) phase. Recent works have shown that simply fine-tuning a pre-trained Vision Transformer (ViT) on new test classes is a strong approach for FSC. Fine-tuning ViTs, however, is expensive in time, compute and storage. This has motivated the design of parameter efficient fine-tuning (PEFT) methods which fine-tune only a fraction of the Transformer's parameters.
  arXiv  Detail & Related papers  (2023-04-04T16:14:39Z)
• Masked Images Are Counterfactual Samples for Robust Fine-tuning [77.82348472169335]
  Fine-tuning deep learning models can lead to a trade-off between in-distribution (ID) performance and out-of-distribution (OOD) robustness. We propose a novel fine-tuning method, which uses masked images as counterfactual samples that help improve the robustness of the fine-tuning model.
  arXiv  Detail & Related papers  (2023-03-06T11:51:28Z)
• Auto-FedRL: Federated Hyperparameter Optimization for Multi-institutional Medical Image Segmentation [48.821062916381685]
  Federated learning (FL) is a distributed machine learning technique that enables collaborative model training while avoiding explicit data sharing. In this work, we propose an efficient reinforcement learning(RL)-based federated hyperparameter optimization algorithm, termed Auto-FedRL. The effectiveness of the proposed method is validated on a heterogeneous data split of the CIFAR-10 dataset and two real-world medical image segmentation datasets.
  arXiv  Detail & Related papers  (2022-03-12T04:11:42Z)
• About Explicit Variance Minimization: Training Neural Networks for Medical Imaging With Limited Data Annotations [2.3204178451683264]
  Variance Aware Training (VAT) method exploits this property by introducing the variance error into the model loss function. We validate VAT on three medical imaging datasets from diverse domains and various learning objectives.
  arXiv  Detail & Related papers  (2021-05-28T21:34:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.