Byzantine-resilient Decentralized Stochastic Gradient Descent

• URL: http://arxiv.org/abs/2002.08569v4
• Date: Wed, 20 Oct 2021 09:07:16 GMT
• Title: Byzantine-resilient Decentralized Stochastic Gradient Descent
• Authors: Shangwei Guo, Tianwei Zhang, Han Yu, Xiaofei Xie, Lei Ma, Tao Xiang, and Yang Liu
• Abstract summary: We present an in-depth study towards the Byzantine resilience of decentralized learning systems. We propose UBAR, a novel algorithm to enhance decentralized learning with Byzantine Fault Tolerance.
• Score: 85.15773446094576
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Decentralized learning has gained great popularity to improve learning efficiency and preserve data privacy. Each computing node makes equal contribution to collaboratively learn a Deep Learning model. The elimination of centralized Parameter Servers (PS) can effectively address many issues such as privacy, performance bottleneck and single-point-failure. However, how to achieve Byzantine Fault Tolerance in decentralized learning systems is rarely explored, although this problem has been extensively studied in centralized systems. In this paper, we present an in-depth study towards the Byzantine resilience of decentralized learning systems with two contributions. First, from the adversarial perspective, we theoretically illustrate that Byzantine attacks are more dangerous and feasible in decentralized learning systems: even one malicious participant can arbitrarily alter the models of other participants by sending carefully crafted updates to its neighbors. Second, from the defense perspective, we propose UBAR, a novel algorithm to enhance decentralized learning with Byzantine Fault Tolerance. Specifically, UBAR provides a Uniform Byzantine-resilient Aggregation Rule for benign nodes to select the useful parameter updates and filter out the malicious ones in each training iteration. It guarantees that each benign node in a decentralized system can train a correct model under very strong Byzantine attacks with an arbitrary number of faulty nodes. We conduct extensive experiments on standard image classification tasks and the results indicate that UBAR can effectively defeat both simple and sophisticated Byzantine attacks with higher performance efficiency than existing solutions.

Related papers

• Byzantine-Robust Aggregation for Securing Decentralized Federated Learning [0.32985979395737774]
  Federated Learning (FL) emerges as a distributed machine learning approach that addresses privacy concerns by training AI models locally on devices. Decentralized Federated Learning (DFL) extends the FL paradigm by eliminating the central server, thereby enhancing scalability and robustness through the avoidance of a single point of failure. We present a novel Byzantine-robust aggregation algorithm to enhance the security of DFL environments, coined WFAgg.
  arXiv  Detail & Related papers  (2024-09-26T11:36:08Z)
• Can Decentralized Learning be more robust than Federated Learning? [8.873449722727026]
  We introduce two textitnew attacks against Decentralized Learning (DL) We demonstrate our attacks' efficiency against Self-Centered Clipping, the state-of---the-art robust DL protocol. We show that the capabilities decentralization grants to Byzantine users result in decentralized learning emphalways providing less robustness than federated learning.
  arXiv  Detail & Related papers  (2023-03-07T11:53:37Z)
• Robust Distributed Learning Against Both Distributional Shifts and Byzantine Attacks [29.34471516011148]
  In distributed learning systems, issues may arise from two sources. On one hand, due to distributional shifts between training data and test data, the model could exhibit poor out-of-sample performance. On the other hand, a portion of trained nodes might be subject to byzantine attacks which could invalidate the model.
  arXiv  Detail & Related papers  (2022-10-29T20:08:07Z)
• Does Decentralized Learning with Non-IID Unlabeled Data Benefit from Self Supervision? [51.00034621304361]
  We study decentralized learning with unlabeled data through the lens of self-supervised learning (SSL) We study the effectiveness of contrastive learning algorithms under decentralized learning settings.
  arXiv  Detail & Related papers  (2022-10-20T01:32:41Z)
• On the (In)security of Peer-to-Peer Decentralized Machine Learning [16.671864590599288]
  We introduce a suite of novel attacks for both passive and active decentralized adversaries. We demonstrate that, contrary to what is claimed by decentralized learning proposers, decentralized learning does not offer any security advantage over federated learning.
  arXiv  Detail & Related papers  (2022-05-17T15:36:50Z)
• Byzantine-Robust Decentralized Learning via ClippedGossip [61.03711813598128]
  We propose a ClippedGossip algorithm for Byzantine-robust consensus optimization. We demonstrate the encouraging empirical performance of ClippedGossip under a large number of attacks.
  arXiv  Detail & Related papers  (2022-02-03T12:04:36Z)
• Hybrid Dynamic Contrast and Probability Distillation for Unsupervised Person Re-Id [109.1730454118532]
  Unsupervised person re-identification (Re-Id) has attracted increasing attention due to its practical application in the read-world video surveillance system. We present the hybrid dynamic cluster contrast and probability distillation algorithm. It formulates the unsupervised Re-Id problem into an unified local-to-global dynamic contrastive learning and self-supervised probability distillation framework.
  arXiv  Detail & Related papers  (2021-09-29T02:56:45Z)
• Secure Distributed Training at Scale [65.7538150168154]
  Training in presence of peers requires specialized distributed training algorithms with Byzantine tolerance. We propose a novel protocol for secure (Byzantine-tolerant) decentralized training that emphasizes communication efficiency.
  arXiv  Detail & Related papers  (2021-06-21T17:00:42Z)
• Byzantine-Robust Learning on Heterogeneous Datasets via Bucketing [55.012801269326594]
  In Byzantine robust distributed learning, a central server wants to train a machine learning model over data distributed across multiple workers. A fraction of these workers may deviate from the prescribed algorithm and send arbitrary messages. We propose a simple bucketing scheme that adapts existing robust algorithms to heterogeneous datasets at a negligible computational cost.
  arXiv  Detail & Related papers  (2020-06-16T17:58:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.