Understanding Byzantine Robustness in Federated Learning with A Black-box Server

• URL: http://arxiv.org/abs/2408.06042v1
• Date: Mon, 12 Aug 2024 10:18:24 GMT
• Title: Understanding Byzantine Robustness in Federated Learning with A Black-box Server
• Authors: Fangyuan Zhao, Yuexiang Xie, Xuebin Ren, Bolin Ding, Shusen Yang, Yaliang Li,
• Abstract summary: Federated learning (FL) becomes vulnerable to Byzantine attacks where some of participators tend to damage the utility or discourage the convergence of the learned model. Previous works propose to apply robust rules to aggregate updates from participators against different types of Byzantine attacks. In practice, FL systems can involve a black-box server that makes the adopted aggregation rule inaccessible to participants, which can naturally defend or weaken some Byzantine attacks.
• Score: 47.98788470469994
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Federated learning (FL) becomes vulnerable to Byzantine attacks where some of participators tend to damage the utility or discourage the convergence of the learned model via sending their malicious model updates. Previous works propose to apply robust rules to aggregate updates from participators against different types of Byzantine attacks, while at the same time, attackers can further design advanced Byzantine attack algorithms targeting specific aggregation rule when it is known. In practice, FL systems can involve a black-box server that makes the adopted aggregation rule inaccessible to participants, which can naturally defend or weaken some Byzantine attacks. In this paper, we provide an in-depth understanding on the Byzantine robustness of the FL system with a black-box server. Our investigation demonstrates the improved Byzantine robustness of a black-box server employing a dynamic defense strategy. We provide both empirical evidence and theoretical analysis to reveal that the black-box server can mitigate the worst-case attack impact from a maximum level to an expectation level, which is attributed to the inherent inaccessibility and randomness offered by a black-box server.The source code is available at https://github.com/alibaba/FederatedScope/tree/Byzantine_attack_defense to promote further research in the community.

Related papers

• Do We Really Need to Design New Byzantine-robust Aggregation Rules? [9.709243052112921]
  Federated learning (FL) allows multiple clients to collaboratively train a global machine learning model through a server. The decentralized aspect of FL makes it susceptible to poisoning attacks, where malicious clients can manipulate the global model. We present FoundationFL, a novel defense mechanism against poisoning attacks.
  arXiv  Detail & Related papers  (2025-01-29T02:28:03Z)
• Rethinking Byzantine Robustness in Federated Recommendation from Sparse Aggregation Perspective [65.65471972217814]
  federated recommendation (FR) based on federated learning (FL) emerges, keeping the personal data on the local client and updating a model collaboratively. FR has a unique sparse aggregation mechanism, where the embedding of each item is updated by only partial clients, instead of full clients in a dense aggregation of general FL. In this paper, we reformulate the Byzantine robustness under sparse aggregation by defining the aggregation for a single item as the smallest execution unit. We propose a family of effective attack strategies, named Spattack, which exploit the vulnerability in sparse aggregation and are categorized along the adversary's knowledge and capability.
  arXiv  Detail & Related papers  (2025-01-06T15:19:26Z)
• BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learning [85.2564206440109]
  This paper reveals the threats in this practical scenario that backdoor attacks can remain effective even after defenses. We introduce the emphtoolns attack, which is resistant to backdoor detection and model fine-tuning defenses.
  arXiv  Detail & Related papers  (2023-11-20T02:21:49Z)
• Certifiable Black-Box Attacks with Randomized Adversarial Examples: Breaking Defenses with Provable Confidence [34.35162562625252]
  Black-box adversarial attacks have demonstrated strong potential to compromise machine learning models. We study a new paradigm of black-box attacks with provable guarantees. This new black-box attack unveils significant vulnerabilities of machine learning models.
  arXiv  Detail & Related papers  (2023-04-10T01:12:09Z)
• Byzantines can also Learn from History: Fall of Centered Clipping in Federated Learning [6.974212769362569]
  In this work, we introduce a novel attack strategy that can circumvent the defences of CC framework. We also propose a new robust and fast defence mechanism that is effective against the proposed and other existing Byzantine attacks.
  arXiv  Detail & Related papers  (2022-08-21T14:39:30Z)
• Certified Federated Adversarial Training [3.474871319204387]
  We tackle the scenario of securing FL systems conducting adversarial training when a quorum of workers could be completely malicious. We model an attacker who poisons the model to insert a weakness into the adversarial training such that the model displays apparent adversarial robustness. We show that this defence can preserve adversarial robustness even against an adaptive attacker.
  arXiv  Detail & Related papers  (2021-12-20T13:40:20Z)
• Fixed Points in Cyber Space: Rethinking Optimal Evasion Attacks in the Age of AI-NIDS [70.60975663021952]
  We study blackbox adversarial attacks on network classifiers. We argue that attacker-defender fixed points are themselves general-sum games with complex phase transitions. We show that a continual learning approach is required to study attacker-defender dynamics.
  arXiv  Detail & Related papers  (2021-11-23T23:42:16Z)
• Adversarial training may be a double-edged sword [50.09831237090801]
  We show that some geometric consequences of adversarial training on the decision boundary of deep networks give an edge to certain types of black-box attacks. In particular, we define a metric called robustness gain to show that while adversarial training is an effective method to dramatically improve the robustness in white-box scenarios, it may not provide such a good robustness gain against the more realistic decision-based black-box attacks.
  arXiv  Detail & Related papers  (2021-07-24T19:09:16Z)
• Spanning Attack: Reinforce Black-box Attacks with Unlabeled Data [96.92837098305898]
  Black-box attacks aim to craft adversarial perturbations by querying input-output pairs of machine learning models. Black-box attacks often suffer from the issue of query inefficiency due to the high dimensionality of the input space. We propose a novel technique called the spanning attack, which constrains adversarial perturbations in a low-dimensional subspace via spanning an auxiliary unlabeled dataset.
  arXiv  Detail & Related papers  (2020-05-11T05:57:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.