Exploring the Robustness of Decentralized Training for Large Language Models

• URL: http://arxiv.org/abs/2312.00843v1
• Date: Fri, 1 Dec 2023 04:04:03 GMT
• Title: Exploring the Robustness of Decentralized Training for Large Language Models
• Authors: Lin Lu, Chenxi Dai, Wangcheng Tao, Binhang Yuan, Yanan Sun, Pan Zhou
• Abstract summary: Decentralized training of large language models has emerged as an effective way to democratize this technology. This paper explores the robustness of decentralized training from three main perspectives.
• Score: 51.41850749014054
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Decentralized training of large language models has emerged as an effective way to democratize this technology. However, the potential threats associated with this approach have not been carefully discussed, which would hinder the development of decentralized training infrastructures. This paper aims to initiate discussion towards this end by exploring the robustness of decentralized training from three main perspectives. First, we demonstrate the vulnerabilities inherent in decentralized training frameworks in terms of hardware, data, and models. Second, we highlight the fundamental difference between decentralized foundation model training and vanilla federated learning, where the security techniques employed in federated learning cannot be applied directly. Third, we discuss the essential components required for a robust and efficient decentralized training framework and present a case study by modeling a concrete threat model. Our objective in this vision paper is to emphasize the importance of addressing security concerns in the context of decentralized training for large language models.

Related papers

• FEDLAD: Federated Evaluation of Deep Leakage Attacks and Defenses [50.921333548391345]
  Federated Learning is a privacy preserving decentralized machine learning paradigm. Recent research has revealed that private ground truth data can be recovered through a gradient technique known as Deep Leakage. This paper introduces the FEDLAD Framework (Federated Evaluation of Deep Leakage Attacks and Defenses), a comprehensive benchmark for evaluating Deep Leakage attacks and defenses.
  arXiv  Detail & Related papers  (2024-11-05T11:42:26Z)
• Byzantine-Robust Aggregation for Securing Decentralized Federated Learning [0.32985979395737774]
  Federated Learning (FL) emerges as a distributed machine learning approach that addresses privacy concerns by training AI models locally on devices. Decentralized Federated Learning (DFL) extends the FL paradigm by eliminating the central server, thereby enhancing scalability and robustness through the avoidance of a single point of failure. We present a novel Byzantine-robust aggregation algorithm to enhance the security of DFL environments, coined WFAgg.
  arXiv  Detail & Related papers  (2024-09-26T11:36:08Z)
• A Trustworthy AIoT-enabled Localization System via Federated Learning and Blockchain [29.968086297894626]
  We propose a framework named DFLoc to achieve precise 3D localization tasks. Specifically, we address the issue of single-point failure for a reliable and accurate indoor localization system. We introduce an updated model verification mechanism within the blockchain to alleviate the concern of malicious node attacks.
  arXiv  Detail & Related papers  (2024-07-08T04:14:19Z)
• Mitigating Communications Threats in Decentralized Federated Learning through Moving Target Defense [0.0]
  Decentralized Federated Learning (DFL) has enabled the training of machine learning models across federated participants. This paper introduces a security module to counter communication-based attacks for DFL platforms. The effectiveness of the security module is validated through experiments with the MNIST dataset and eclipse attacks.
  arXiv  Detail & Related papers  (2023-07-21T17:43:50Z)
• Networked Communication for Decentralised Agents in Mean-Field Games [59.01527054553122]
  We introduce networked communication to the mean-field game framework. We prove that our architecture has sample guarantees bounded between those of the centralised- and independent-learning cases.
  arXiv  Detail & Related papers  (2023-06-05T10:45:39Z)
• On the (In)security of Peer-to-Peer Decentralized Machine Learning [16.671864590599288]
  We introduce a suite of novel attacks for both passive and active decentralized adversaries. We demonstrate that, contrary to what is claimed by decentralized learning proposers, decentralized learning does not offer any security advantage over federated learning.
  arXiv  Detail & Related papers  (2022-05-17T15:36:50Z)
• Secure Distributed Training at Scale [65.7538150168154]
  Training in presence of peers requires specialized distributed training algorithms with Byzantine tolerance. We propose a novel protocol for secure (Byzantine-tolerant) decentralized training that emphasizes communication efficiency.
  arXiv  Detail & Related papers  (2021-06-21T17:00:42Z)
• Consensus Control for Decentralized Deep Learning [72.50487751271069]
  Decentralized training of deep learning models enables on-device learning over networks, as well as efficient scaling to large compute clusters. We show in theory that when the training consensus distance is lower than a critical quantity, decentralized training converges as fast as the centralized counterpart. Our empirical insights allow the principled design of better decentralized training schemes that mitigate the performance drop.
  arXiv  Detail & Related papers  (2021-02-09T13:58:33Z)
• Decentralized Federated Learning Preserves Model and Data Privacy [77.454688257702]
  We propose a fully decentralized approach, which allows to share knowledge between trained models. Students are trained on the output of their teachers via synthetically generated input data. The results show that an untrained student model, trained on the teachers output reaches comparable F1-scores as the teacher.
  arXiv  Detail & Related papers  (2021-02-01T14:38:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.