Tunable Soft Prompts are Messengers in Federated Learning
- URL: http://arxiv.org/abs/2311.06805v1
- Date: Sun, 12 Nov 2023 11:01:10 GMT
- Title: Tunable Soft Prompts are Messengers in Federated Learning
- Authors: Chenhe Dong, Yuexiang Xie, Bolin Ding, Ying Shen, Yaliang Li
- Abstract summary: Federated learning (FL) enables multiple participants to collaboratively train machine learning models using decentralized data sources.
The lack of model privacy protection in FL becomes an unneglectable challenge.
We propose a novel FL training approach that accomplishes information exchange among participants via tunable soft prompts.
- Score: 55.924749085481544
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Federated learning (FL) enables multiple participants to collaboratively
train machine learning models using decentralized data sources, alleviating
privacy concerns that arise from directly sharing local data. However, the lack
of model privacy protection in FL becomes an unneglectable challenge,
especially when people want to federally finetune models based on a proprietary
large language model. In this study, we propose a novel FL training approach
that accomplishes information exchange among participants via tunable soft
prompts. These soft prompts, updated and transmitted between the server and
clients, assume the role of the global model parameters and serve as messengers
to deliver useful knowledge from the local data and global model. As the global
model itself is not required to be shared and the local training is conducted
based on an auxiliary model with fewer parameters than the global model, the
proposed approach provides protection for the global model while reducing
communication and computation costs in FL. Extensive experiments show the
effectiveness of the proposed approach compared to several baselines. We have
released the source code at
\url{https://github.com/alibaba/FederatedScope/tree/fedsp/federatedscope/nlp/fedsp}.
Related papers
- Rethinking Client Drift in Federated Learning: A Logit Perspective [125.35844582366441]
Federated Learning (FL) enables multiple clients to collaboratively learn in a distributed way, allowing for privacy protection.
We find that the difference in logits between the local and global models increases as the model is continuously updated.
We propose a new algorithm, named FedCSD, a Class prototype Similarity Distillation in a federated framework to align the local and global models.
arXiv Detail & Related papers (2023-08-20T04:41:01Z) - FedSoup: Improving Generalization and Personalization in Federated
Learning via Selective Model Interpolation [32.36334319329364]
Cross-silo federated learning (FL) enables the development of machine learning models on datasets distributed across data centers.
Recent research has found that current FL algorithms face a trade-off between local and global performance when confronted with distribution shifts.
We propose a novel federated model soup method to optimize the trade-off between local and global performance.
arXiv Detail & Related papers (2023-07-20T00:07:29Z) - Personalized Federated Learning with Hidden Information on Personalized
Prior [18.8426865970643]
We propose pFedBreD, a framework to solve the problem we model using Bregman divergence regularization.
Our experiments show that our proposal significantly outcompetes other PFL algorithms on multiple public benchmarks.
arXiv Detail & Related papers (2022-11-19T12:45:19Z) - Fine-tuning Global Model via Data-Free Knowledge Distillation for
Non-IID Federated Learning [86.59588262014456]
Federated Learning (FL) is an emerging distributed learning paradigm under privacy constraint.
We propose a data-free knowledge distillation method to fine-tune the global model in the server (FedFTG)
Our FedFTG significantly outperforms the state-of-the-art (SOTA) FL algorithms and can serve as a strong plugin for enhancing FedAvg, FedProx, FedDyn, and SCAFFOLD.
arXiv Detail & Related papers (2022-03-17T11:18:17Z) - Turning Federated Learning Systems Into Covert Channels [3.7168491743915646]
Federated learning (FL) goes beyond traditional, centralized machine learning by distributing model training among edge clients.
In this paper, we put forward a novel attacker model aiming at turning FL systems into covert channels to implement a stealth communication infrastructure.
arXiv Detail & Related papers (2021-04-21T14:32:03Z) - WAFFLe: Weight Anonymized Factorization for Federated Learning [88.44939168851721]
In domains where data are sensitive or private, there is great value in methods that can learn in a distributed manner without the data ever leaving the local devices.
We propose Weight Anonymized Factorization for Federated Learning (WAFFLe), an approach that combines the Indian Buffet Process with a shared dictionary of weight factors for neural networks.
arXiv Detail & Related papers (2020-08-13T04:26:31Z) - Federated Learning With Quantized Global Model Updates [84.55126371346452]
We study federated learning, which enables mobile devices to utilize their local datasets to train a global model.
We introduce a lossy FL (LFL) algorithm, in which both the global model and the local model updates are quantized before being transmitted.
arXiv Detail & Related papers (2020-06-18T16:55:20Z) - Continual Local Training for Better Initialization of Federated Models [14.289213162030816]
Federated learning (FL) refers to the learning paradigm that trains machine learning models directly in decentralized systems.
The popular FL algorithm emphFederated Averaging (FedAvg) suffers from weight divergence.
We propose the local continual training strategy to address this problem.
arXiv Detail & Related papers (2020-05-26T12:27:31Z) - Think Locally, Act Globally: Federated Learning with Local and Global
Representations [92.68484710504666]
Federated learning is a method of training models on private data distributed over multiple devices.
We propose a new federated learning algorithm that jointly learns compact local representations on each device.
We also evaluate on the task of personalized mood prediction from real-world mobile data where privacy is key.
arXiv Detail & Related papers (2020-01-06T12:40:21Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.