Federated Adaptive Prompt Tuning for Multi-Domain Collaborative Learning

• URL: http://arxiv.org/abs/2211.07864v4
• Date: Tue, 12 Mar 2024 05:23:54 GMT
• Title: Federated Adaptive Prompt Tuning for Multi-Domain Collaborative Learning
• Authors: Shangchao Su and Mingzhao Yang and Bin Li and Xiangyang Xue
• Abstract summary: Federated learning (FL) enables multiple clients to collaboratively train a global model without disclosing their data. We propose a federated adaptive prompt tuning algorithm, FedAPT, for multi-domain collaborative image classification.
• Score: 44.604485649167216
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Federated learning (FL) enables multiple clients to collaboratively train a global model without disclosing their data. Previous researches often require training the complete model parameters. However, the emergence of powerful pre-trained models makes it possible to achieve higher performance with fewer learnable parameters in FL. In this paper, we propose a federated adaptive prompt tuning algorithm, FedAPT, for multi-domain collaborative image classification with powerful foundation models, like CLIP. Compared with direct federated prompt tuning, our core idea is to adaptively unlock specific domain knowledge for each test sample in order to provide them with personalized prompts. To implement this idea, we design an adaptive prompt tuning module, which consists of a meta prompt, an adaptive network, and some keys. The server randomly generates a set of keys and assigns a unique key to each client. Then all clients cooperatively train the global adaptive network and meta prompt with the local datasets and the frozen keys. Ultimately, the global aggregation model can assign a personalized prompt to CLIP based on the domain features of each test sample. We perform extensive experiments on two multi-domain image classification datasets across two different settings -- supervised and unsupervised. The results show that FedAPT can achieve better performance with less than 10\% of the number of parameters of the fully trained model, and the global model can perform well in diverse client domains simultaneously. The source code is available at \url{https://github.com/leondada/FedAPT}.

Related papers

• Collaborative and Efficient Personalization with Mixtures of Adaptors [5.195669033269619]
  We propose a parameter-efficient framework to tackle multi-task learning problems. We call our framework Federated Low-Rank Adaptive Learning (FLoRAL) We show promising experimental results on synthetic datasets and real-world federated multi-task problems.
  arXiv  Detail & Related papers  (2024-10-04T15:11:15Z)
• Multi-Level Additive Modeling for Structured Non-IID Federated Learning [54.53672323071204]
  We train models organized in a multi-level structure, called Multi-level Additive Models (MAM)'', for better knowledge-sharing across heterogeneous clients. In federated MAM (FeMAM), each client is assigned to at most one model per level and its personalized prediction sums up the outputs of models assigned to it across all levels. Experiments show that FeMAM surpasses existing clustered FL and personalized FL methods in various non-IID settings.
  arXiv  Detail & Related papers  (2024-05-26T07:54:53Z)
• Communication-Efficient Multimodal Federated Learning: Joint Modality and Client Selection [14.261582708240407]
  Multimodal Federated learning (FL) aims to enrich model training in FL settings where clients are collecting measurements across multiple modalities. Key challenges to multimodal FL remain unaddressed, particularly in heterogeneous network settings. We propose mmFedMC, a new FL methodology that can tackle the above-mentioned challenges in multimodal settings.
  arXiv  Detail & Related papers  (2024-01-30T02:16:19Z)
• FedRA: A Random Allocation Strategy for Federated Tuning to Unleash the Power of Heterogeneous Clients [50.13097183691517]
  In real-world federated scenarios, there often exist a multitude of heterogeneous clients with varying computation and communication resources. We propose a novel federated tuning algorithm, FedRA. In each communication round, FedRA randomly generates an allocation matrix. It reorganizes a small number of layers from the original model based on the allocation matrix and fine-tunes using adapters.
  arXiv  Detail & Related papers  (2023-11-19T04:43:16Z)
• Adaptive Test-Time Personalization for Federated Learning [51.25437606915392]
  We introduce a novel setting called test-time personalized federated learning (TTPFL) In TTPFL, clients locally adapt a global model in an unsupervised way without relying on any labeled data during test-time. We propose a novel algorithm called ATP to adaptively learn the adaptation rates for each module in the model from distribution shifts among source domains.
  arXiv  Detail & Related papers  (2023-10-28T20:42:47Z)
• Prototype Helps Federated Learning: Towards Faster Convergence [38.517903009319994]
  Federated learning (FL) is a distributed machine learning technique in which multiple clients cooperate to train a shared model without exchanging their raw data. In this paper, a prototype-based federated learning framework is proposed, which can achieve better inference performance with only a few changes to the last global iteration of the typical federated learning process.
  arXiv  Detail & Related papers  (2023-03-22T04:06:29Z)
• Optimizing Server-side Aggregation For Robust Federated Learning via Subspace Training [80.03567604524268]
  Non-IID data distribution across clients and poisoning attacks are two main challenges in real-world federated learning systems. We propose SmartFL, a generic approach that optimize the server-side aggregation process. We provide theoretical analyses of the convergence and generalization capacity for SmartFL.
  arXiv  Detail & Related papers  (2022-11-10T13:20:56Z)
• LPT: Long-tailed Prompt Tuning for Image Classification [178.52948452353834]
  We introduce several trainable prompts into a frozen pretrained model to adapt it to long-tailed data. In phase 1, we train the shared prompt via supervised prompt tuning to adapt a pretrained model to the desired long-tailed domain. In phase 2, we use the learnt shared prompt as query to select a small best matched set for a group of similar samples.
  arXiv  Detail & Related papers  (2022-10-03T15:47:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.