Federated Reinforcement Learning: Linear Speedup Under Markovian
Sampling
- URL: http://arxiv.org/abs/2206.10185v1
- Date: Tue, 21 Jun 2022 08:39:12 GMT
- Title: Federated Reinforcement Learning: Linear Speedup Under Markovian
Sampling
- Authors: Sajad Khodadadian, Pranay Sharma, Gauri Joshi, Siva Theja Maguluri
- Abstract summary: We consider a federated reinforcement learning framework where multiple agents collaboratively learn a global model.
We propose federated versions of on-policy TD, off-policy TD and Q-learning, and analyze their convergence.
We are the first to consider Markovian noise and multiple local updates, and prove a linear convergence speedup with respect to the number of agents.
- Score: 17.943014287720395
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Since reinforcement learning algorithms are notoriously data-intensive, the
task of sampling observations from the environment is usually split across
multiple agents. However, transferring these observations from the agents to a
central location can be prohibitively expensive in terms of the communication
cost, and it can also compromise the privacy of each agent's local behavior
policy. In this paper, we consider a federated reinforcement learning framework
where multiple agents collaboratively learn a global model, without sharing
their individual data and policies. Each agent maintains a local copy of the
model and updates it using locally sampled data. Although having N agents
enables the sampling of N times more data, it is not clear if it leads to
proportional convergence speedup. We propose federated versions of on-policy
TD, off-policy TD and Q-learning, and analyze their convergence. For all these
algorithms, to the best of our knowledge, we are the first to consider
Markovian noise and multiple local updates, and prove a linear convergence
speedup with respect to the number of agents. To obtain these results, we show
that federated TD and Q-learning are special cases of a general framework for
federated stochastic approximation with Markovian noise, and we leverage this
framework to provide a unified convergence analysis that applies to all the
algorithms.
Related papers
- PeFAD: A Parameter-Efficient Federated Framework for Time Series Anomaly Detection [51.20479454379662]
We propose a.
Federated Anomaly Detection framework named PeFAD with the increasing privacy concerns.
We conduct extensive evaluations on four real datasets, where PeFAD outperforms existing state-of-the-art baselines by up to 28.74%.
arXiv Detail & Related papers (2024-06-04T13:51:08Z) - Asynchronous Diffusion Learning with Agent Subsampling and Local Updates [47.25856291277345]
We investigate a network of agents operating asynchronously, aiming to discover an ideal global model that suits individual local datasets.
We prove that the resulting asynchronous diffusion strategy is stable in the mean-square error sense.
arXiv Detail & Related papers (2024-02-08T10:07:30Z) - FedLPA: One-shot Federated Learning with Layer-Wise Posterior Aggregation [7.052566906745796]
FedLPA is a layer-wise posterior aggregation method for federated learning.
We show that FedLPA significantly improves learning performance over state-of-the-art methods across several metrics.
arXiv Detail & Related papers (2023-09-30T10:51:27Z) - The Blessing of Heterogeneity in Federated Q-Learning: Linear Speedup
and Beyond [44.43850105124659]
We consider federated Q-learning, which aims to learn an optimal Q-function by periodically aggregating local Q-estimates trained on local data alone.
We provide sample complexity guarantees for both the synchronous and asynchronous variants of federated Q-learning.
We propose a novel federated Q-learning algorithm with importance averaging, giving larger weights to more frequently visited state-action pairs.
arXiv Detail & Related papers (2023-05-18T04:18:59Z) - FedAvg with Fine Tuning: Local Updates Lead to Representation Learning [54.65133770989836]
Federated Averaging (FedAvg) algorithm consists of alternating between a few local gradient updates at client nodes, followed by a model averaging update at the server.
We show that the reason behind generalizability of the FedAvg's output is its power in learning the common data representation among the clients' tasks.
We also provide empirical evidence demonstrating FedAvg's representation learning ability in federated image classification with heterogeneous data.
arXiv Detail & Related papers (2022-05-27T00:55:24Z) - Speeding up Heterogeneous Federated Learning with Sequentially Trained
Superclients [19.496278017418113]
Federated Learning (FL) allows training machine learning models in privacy-constrained scenarios by enabling the cooperation of edge devices without requiring local data sharing.
This approach raises several challenges due to the different statistical distribution of the local datasets and the clients' computational heterogeneity.
We propose FedSeq, a novel framework leveraging the sequential training of subgroups of heterogeneous clients, i.e. superclients, to emulate the centralized paradigm in a privacy-compliant way.
arXiv Detail & Related papers (2022-01-26T12:33:23Z) - Fully Distributed Actor-Critic Architecture for Multitask Deep
Reinforcement Learning [6.628062414583634]
We propose a fully distributed actor-critic architecture, named Diff-DAC, with application to multitask reinforcement learning (MRL)
Agents communicate their value and policy parameters to their neighbours, diffusing the information across a network of agents with no need for a central station.
We prove almost sure convergence of Diff-DAC to a common policy under general assumptions that hold even for deep-neural network approximations.
arXiv Detail & Related papers (2021-10-23T21:57:43Z) - Exploiting Shared Representations for Personalized Federated Learning [54.65133770989836]
We propose a novel federated learning framework and algorithm for learning a shared data representation across clients and unique local heads for each client.
Our algorithm harnesses the distributed computational power across clients to perform many local-updates with respect to the low-dimensional local parameters for every update of the representation.
This result is of interest beyond federated learning to a broad class of problems in which we aim to learn a shared low-dimensional representation among data distributions.
arXiv Detail & Related papers (2021-02-14T05:36:25Z) - Distributed Q-Learning with State Tracking for Multi-agent Networked
Control [61.63442612938345]
This paper studies distributed Q-learning for Linear Quadratic Regulator (LQR) in a multi-agent network.
We devise a state tracking (ST) based Q-learning algorithm to design optimal controllers for agents.
arXiv Detail & Related papers (2020-12-22T22:03:49Z) - Dynamic Federated Learning [57.14673504239551]
Federated learning has emerged as an umbrella term for centralized coordination strategies in multi-agent environments.
We consider a federated learning model where at every iteration, a random subset of available agents perform local updates based on their data.
Under a non-stationary random walk model on the true minimizer for the aggregate optimization problem, we establish that the performance of the architecture is determined by three factors, namely, the data variability at each agent, the model variability across all agents, and a tracking term that is inversely proportional to the learning rate of the algorithm.
arXiv Detail & Related papers (2020-02-20T15:00:54Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.