Compressed Federated Reinforcement Learning with a Generative Model

• URL: http://arxiv.org/abs/2404.10635v6
• Date: Mon, 14 Oct 2024 16:11:57 GMT
• Title: Compressed Federated Reinforcement Learning with a Generative Model
• Authors: Ali Beikmohammadi, Sarit Khirirat, Sindri Magnússon,
• Abstract summary: Reinforcement learning has recently gained unprecedented popularity, yet it still grapples with sample inefficiency. Addressing this challenge, federated reinforcement learning (FedRL) has emerged, wherein agents collaboratively learn a single policy by aggregating local estimations. We propose CompFedRL, a communication-efficient FedRL approach incorporating both textitperiodic aggregation and (direct/error-feedback) compression mechanisms.
• Score: 11.074080383657453
• License:
• Abstract: Reinforcement learning has recently gained unprecedented popularity, yet it still grapples with sample inefficiency. Addressing this challenge, federated reinforcement learning (FedRL) has emerged, wherein agents collaboratively learn a single policy by aggregating local estimations. However, this aggregation step incurs significant communication costs. In this paper, we propose CompFedRL, a communication-efficient FedRL approach incorporating both \textit{periodic aggregation} and (direct/error-feedback) compression mechanisms. Specifically, we consider compressed federated $Q$-learning with a generative model setup, where a central server learns an optimal $Q$-function by periodically aggregating compressed $Q$-estimates from local agents. For the first time, we characterize the impact of these two mechanisms (which have remained elusive) by providing a finite-time analysis of our algorithm, demonstrating strong convergence behaviors when utilizing either direct or error-feedback compression. Our bounds indicate improved solution accuracy concerning the number of agents and other federated hyperparameters while simultaneously reducing communication costs. To corroborate our theory, we also conduct in-depth numerical experiments to verify our findings, considering Top-$K$ and Sparsified-$K$ sparsification operators.

Related papers

• Boosting the Performance of Decentralized Federated Learning via Catalyst Acceleration [66.43954501171292]
  We introduce Catalyst Acceleration and propose an acceleration Decentralized Federated Learning algorithm called DFedCata. DFedCata consists of two main components: the Moreau envelope function, which addresses parameter inconsistencies, and Nesterov's extrapolation step, which accelerates the aggregation phase. Empirically, we demonstrate the advantages of the proposed algorithm in both convergence speed and generalization performance on CIFAR10/100 with various non-iid data distributions.
  arXiv  Detail & Related papers  (2024-10-09T06:17:16Z)
• Momentum-Based Federated Reinforcement Learning with Interaction and Communication Efficiency [16.002770483584694]
  Federated Reinforcement Learning (FRL) has garnered increasing attention. In this paper, we introduce a new FRL algorithm, named $texttMFPO$. We prove that by proper selection of momentum parameters and interaction frequency, $texttMFPO$ can achieve $tildemathcalO(H-1Nepsilon-3/2N)$ and $tmathcalO(ilon-1N)$.
  arXiv  Detail & Related papers  (2024-05-24T03:23:37Z)
• Compressed and Sparse Models for Non-Convex Decentralized Learning [6.14375469212514]
  Frequent model communication is a significant bottleneck to the efficiency of decentralized machine learning. We present Malcom-PSGD, a novel decentralized ML algorithm that combines model sparsity with a gradient gradient. Our method reduces communication costs by approximately $75%$ when compared to the state-of-the-art.
  arXiv  Detail & Related papers  (2023-11-09T21:55:53Z)
• Pure Exploration in Asynchronous Federated Bandits [57.02106627533004]
  We study the federated pure exploration problem of multi-armed bandits and linear bandits, where $M$ agents cooperatively identify the best arm via communicating with the central server. We propose the first asynchronous multi-armed bandit and linear bandit algorithms for pure exploration with fixed confidence.
  arXiv  Detail & Related papers  (2023-10-17T06:04:00Z)
• Momentum Benefits Non-IID Federated Learning Simply and Provably [22.800862422479913]
  Federated learning is a powerful paradigm for large-scale machine learning. FedAvg and SCAFFOLD are two prominent algorithms to address these challenges. This paper explores the utilization of momentum to enhance the performance of FedAvg and SCAFFOLD.
  arXiv  Detail & Related papers  (2023-06-28T18:52:27Z)
• Compressed Regression over Adaptive Networks [58.79251288443156]
  We derive the performance achievable by a network of distributed agents that solve, adaptively and in the presence of communication constraints, a regression problem. We devise an optimized allocation strategy where the parameters necessary for the optimization can be learned online by the agents.
  arXiv  Detail & Related papers  (2023-04-07T13:41:08Z)
• Provably Efficient Multi-Agent Reinforcement Learning with Fully Decentralized Communication [3.5450828190071655]
  Distributed exploration reduces sampling complexity in reinforcement learning. We show that group performance can be significantly improved when each agent uses a decentralized message-passing protocol. We show that incorporating more agents and more information sharing into the group learning scheme speeds up convergence to the optimal policy.
  arXiv  Detail & Related papers  (2021-10-14T14:27:27Z)
• Permutation Compressors for Provably Faster Distributed Nonconvex Optimization [68.8204255655161]
  We show that the MARINA method of Gorbunov et al (2021) can be considered as a state-of-the-art method in terms of theoretical communication complexity. Theory of MARINA to support the theory of potentially em correlated compressors, extends to the method beyond the classical independent compressors setting.
  arXiv  Detail & Related papers  (2021-10-07T09:38:15Z)
• Federated Learning with Unreliable Clients: Performance Analysis and Mechanism Design [76.29738151117583]
  Federated Learning (FL) has become a promising tool for training effective machine learning models among distributed clients. However, low quality models could be uploaded to the aggregator server by unreliable clients, leading to a degradation or even a collapse of training. We model these unreliable behaviors of clients and propose a defensive mechanism to mitigate such a security risk.
  arXiv  Detail & Related papers  (2021-05-10T08:02:27Z)
• Robust Deep Reinforcement Learning through Adversarial Loss [74.20501663956604]
  Recent studies have shown that deep reinforcement learning agents are vulnerable to small adversarial perturbations on the agent's inputs. We propose RADIAL-RL, a principled framework to train reinforcement learning agents with improved robustness against adversarial attacks.
  arXiv  Detail & Related papers  (2020-08-05T07:49:42Z)
• Intermittent Pulling with Local Compensation for Communication-Efficient Federated Learning [20.964434898554344]
  Federated Learning is a powerful machine learning paradigm to train a global model with highly distributed data. A major bottleneck on the performance of distributed SGD is the communication overhead on pushing local and pulling global model. We propose a novel approach named Gradient Pulling Compensation (PRLC) to reduce communication overhead.
  arXiv  Detail & Related papers  (2020-01-22T20:53:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.