Fault-Tolerant Federated Reinforcement Learning with Theoretical Guarantee

• URL: http://arxiv.org/abs/2110.14074v1
• Date: Tue, 26 Oct 2021 23:01:22 GMT
• Title: Fault-Tolerant Federated Reinforcement Learning with Theoretical Guarantee
• Authors: Flint Xiaofeng Fan, Yining Ma, Zhongxiang Dai, Wei Jing, Cheston Tan, Bryan Kian Hsiang Low
• Abstract summary: We propose the first Federated Reinforcement Learning framework that is tolerant to less than half of the participating agents being random system failures or adversarial attackers. All theoretical results are empirically verified on various RL benchmark tasks.
• Score: 25.555844784263236
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The growing literature of Federated Learning (FL) has recently inspired Federated Reinforcement Learning (FRL) to encourage multiple agents to federatively build a better decision-making policy without sharing raw trajectories. Despite its promising applications, existing works on FRL fail to I) provide theoretical analysis on its convergence, and II) account for random system failures and adversarial attacks. Towards this end, we propose the first FRL framework the convergence of which is guaranteed and tolerant to less than half of the participating agents being random system failures or adversarial attackers. We prove that the sample efficiency of the proposed framework is guaranteed to improve with the number of agents and is able to account for such potential failures or attacks. All theoretical results are empirically verified on various RL benchmark tasks.

Related papers

• Multi-Agent Reinforcement Learning from Human Feedback: Data Coverage and Algorithmic Techniques [65.55451717632317]
  We study Multi-Agent Reinforcement Learning from Human Feedback (MARLHF), exploring both theoretical foundations and empirical validations. We define the task as identifying Nash equilibrium from a preference-only offline dataset in general-sum games. Our findings underscore the multifaceted approach required for MARLHF, paving the way for effective preference-based multi-agent systems.
  arXiv  Detail & Related papers  (2024-09-01T13:14:41Z)
• LiD-FL: Towards List-Decodable Federated Learning [18.89910309677336]
  This paper proposes an algorithmic framework for list-decodable federated learning, where a central server maintains a list of adversaries. Experimental results show that the proposed algorithm can withstand the malicious majority under various attacks.
  arXiv  Detail & Related papers  (2024-08-09T09:29:02Z)
• Decentralized Federated Policy Gradient with Byzantine Fault-Tolerance and Provably Fast Convergence [21.935405256685307]
  In Federated Reinforcement Learning (FRL), agents aim to collaboratively learn a common task, while each agent is acting in its local environment without exchanging raw trajectories. We first propose a new centralized Byzantine fault-tolerant policy (PG) algorithm that improves existing methods by relying only on assumptions standard for non-fault-tolerant PG.
  arXiv  Detail & Related papers  (2024-01-07T14:06:06Z)
• Local Environment Poisoning Attacks on Federated Reinforcement Learning [1.5020330976600738]
  Federated learning (FL) has become a popular tool for solving traditional Reinforcement Learning (RL) tasks. Federated mechanism exposes the system to poisoning by malicious agents that can mislead the trained policy. We propose a general framework to characterize FRL poisoning as an optimization problem and design a poisoning protocol that can be applied to policy-based FRL.
  arXiv  Detail & Related papers  (2023-03-05T17:44:23Z)
• Combating Exacerbated Heterogeneity for Robust Models in Federated Learning [91.88122934924435]
  Combination of adversarial training and federated learning can lead to the undesired robustness deterioration. We propose a novel framework called Slack Federated Adversarial Training (SFAT) We verify the rationality and effectiveness of SFAT on various benchmarked and real-world datasets.
  arXiv  Detail & Related papers  (2023-03-01T06:16:15Z)
• FLIP: A Provable Defense Framework for Backdoor Mitigation in Federated Learning [66.56240101249803]
  We study how hardening benign clients can affect the global model (and the malicious clients) We propose a trigger reverse engineering based defense and show that our method can achieve improvement with guarantee robustness. Our results on eight competing SOTA defense methods show the empirical superiority of our method on both single-shot and continuous FL backdoor attacks.
  arXiv  Detail & Related papers  (2022-10-23T22:24:03Z)
• Fair Federated Learning via Bounded Group Loss [37.72259706322158]
  We propose a general framework for provably fair federated learning. We extend the notion of Bounded Group Loss as a theoretically-grounded approach for group fairness. We provide convergence guarantees for the method as well as fairness guarantees for the resulting solution.
  arXiv  Detail & Related papers  (2022-03-18T23:11:54Z)
• COPA: Certifying Robust Policies for Offline Reinforcement Learning against Poisoning Attacks [49.15885037760725]
  We focus on certifying the robustness of offline reinforcement learning (RL) in the presence of poisoning attacks. We propose the first certification framework, COPA, to certify the number of poisoning trajectories that can be tolerated. We prove that some of the proposed certification methods are theoretically tight and some are NP-Complete problems.
  arXiv  Detail & Related papers  (2022-03-16T05:02:47Z)
• Monotonic Improvement Guarantees under Non-stationarity for Decentralized PPO [66.5384483339413]
  We present a new monotonic improvement guarantee for optimizing decentralized policies in cooperative Multi-Agent Reinforcement Learning (MARL) We show that a trust region constraint can be effectively enforced in a principled way by bounding independent ratios based on the number of agents in training.
  arXiv  Detail & Related papers  (2022-01-31T20:39:48Z)
• Combing Policy Evaluation and Policy Improvement in a Unified f-Divergence Framework [33.90259939664709]
  We study the f-divergence between learning policy and sampling policy and derive a novel DRL framework, termed f-Divergence Reinforcement Learning (FRL) The FRL framework achieves two advantages: (1) policy evaluation and policy improvement processes are derived simultaneously by f-divergence; (2) overestimation issue of value function are alleviated.
  arXiv  Detail & Related papers  (2021-09-24T10:20:46Z)
• Robust Deep Reinforcement Learning through Adversarial Loss [74.20501663956604]
  Recent studies have shown that deep reinforcement learning agents are vulnerable to small adversarial perturbations on the agent's inputs. We propose RADIAL-RL, a principled framework to train reinforcement learning agents with improved robustness against adversarial attacks.
  arXiv  Detail & Related papers  (2020-08-05T07:49:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.