Exploring the Robustness of Decentralized Training for Large Language
Models
- URL: http://arxiv.org/abs/2312.00843v1
- Date: Fri, 1 Dec 2023 04:04:03 GMT
- Title: Exploring the Robustness of Decentralized Training for Large Language
Models
- Authors: Lin Lu, Chenxi Dai, Wangcheng Tao, Binhang Yuan, Yanan Sun, Pan Zhou
- Abstract summary: Decentralized training of large language models has emerged as an effective way to democratize this technology.
This paper explores the robustness of decentralized training from three main perspectives.
- Score: 51.41850749014054
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Decentralized training of large language models has emerged as an effective
way to democratize this technology. However, the potential threats associated
with this approach have not been carefully discussed, which would hinder the
development of decentralized training infrastructures. This paper aims to
initiate discussion towards this end by exploring the robustness of
decentralized training from three main perspectives. First, we demonstrate the
vulnerabilities inherent in decentralized training frameworks in terms of
hardware, data, and models. Second, we highlight the fundamental difference
between decentralized foundation model training and vanilla federated learning,
where the security techniques employed in federated learning cannot be applied
directly. Third, we discuss the essential components required for a robust and
efficient decentralized training framework and present a case study by modeling
a concrete threat model. Our objective in this vision paper is to emphasize the
importance of addressing security concerns in the context of decentralized
training for large language models.
Related papers
- A Trustworthy AIoT-enabled Localization System via Federated Learning and Blockchain [29.968086297894626]
We propose a framework named DFLoc to achieve precise 3D localization tasks.
Specifically, we address the issue of single-point failure for a reliable and accurate indoor localization system.
We introduce an updated model verification mechanism within the blockchain to alleviate the concern of malicious node attacks.
arXiv Detail & Related papers (2024-07-08T04:14:19Z) - Initialisation and Topology Effects in Decentralised Federated Learning [1.5961625979922607]
Decentralised federated learning enables collaborative training of individual machine learning models on distributed devices on a communication network.
This approach enhances data privacy and eliminates both the single point of failure and the necessity for central coordination.
We propose a strategy for uncoordinated initialisation of the artificial neural networks.
arXiv Detail & Related papers (2024-03-23T14:24:36Z) - Mitigating Communications Threats in Decentralized Federated Learning
through Moving Target Defense [0.0]
Decentralized Federated Learning (DFL) has enabled the training of machine learning models across federated participants.
This paper introduces a security module to counter communication-based attacks for DFL platforms.
The effectiveness of the security module is validated through experiments with the MNIST dataset and eclipse attacks.
arXiv Detail & Related papers (2023-07-21T17:43:50Z) - Certified Robustness in Federated Learning [54.03574895808258]
We study the interplay between federated training, personalization, and certified robustness.
We find that the simple federated averaging technique is effective in building not only more accurate, but also more certifiably-robust models.
arXiv Detail & Related papers (2022-06-06T12:10:53Z) - On the (In)security of Peer-to-Peer Decentralized Machine Learning [16.671864590599288]
We introduce a suite of novel attacks for both passive and active decentralized adversaries.
We demonstrate that, contrary to what is claimed by decentralized learning proposers, decentralized learning does not offer any security advantage over federated learning.
arXiv Detail & Related papers (2022-05-17T15:36:50Z) - Secure Distributed Training at Scale [65.7538150168154]
Training in presence of peers requires specialized distributed training algorithms with Byzantine tolerance.
We propose a novel protocol for secure (Byzantine-tolerant) decentralized training that emphasizes communication efficiency.
arXiv Detail & Related papers (2021-06-21T17:00:42Z) - Consensus Control for Decentralized Deep Learning [72.50487751271069]
Decentralized training of deep learning models enables on-device learning over networks, as well as efficient scaling to large compute clusters.
We show in theory that when the training consensus distance is lower than a critical quantity, decentralized training converges as fast as the centralized counterpart.
Our empirical insights allow the principled design of better decentralized training schemes that mitigate the performance drop.
arXiv Detail & Related papers (2021-02-09T13:58:33Z) - Decentralized Federated Learning Preserves Model and Data Privacy [77.454688257702]
We propose a fully decentralized approach, which allows to share knowledge between trained models.
Students are trained on the output of their teachers via synthetically generated input data.
The results show that an untrained student model, trained on the teachers output reaches comparable F1-scores as the teacher.
arXiv Detail & Related papers (2021-02-01T14:38:54Z) - Byzantine-resilient Decentralized Stochastic Gradient Descent [85.15773446094576]
We present an in-depth study towards the Byzantine resilience of decentralized learning systems.
We propose UBAR, a novel algorithm to enhance decentralized learning with Byzantine Fault Tolerance.
arXiv Detail & Related papers (2020-02-20T05:11:04Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.