Addressing Client Drift in Federated Continual Learning with Adaptive Optimization

• URL: http://arxiv.org/abs/2203.13321v1
• Date: Thu, 24 Mar 2022 20:00:03 GMT
• Title: Addressing Client Drift in Federated Continual Learning with Adaptive Optimization
• Authors: Yeshwanth Venkatesha, Youngeun Kim, Hyoungseob Park, Yuhang Li, Priyadarshini Panda
• Abstract summary: We outline a framework for performing Federated Continual Learning (FCL) by using NetTailor as a candidate continual learning approach. We show that adaptive federated optimization can reduce the adverse impact of client drift and showcase its effectiveness on CIFAR100, MiniImagenet, and Decathlon benchmarks.
• Score: 10.303676184878896
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Federated learning has been extensively studied and is the prevalent method for privacy-preserving distributed learning in edge devices. Correspondingly, continual learning is an emerging field targeted towards learning multiple tasks sequentially. However, there is little attention towards additional challenges emerging when federated aggregation is performed in a continual learning system. We identify \textit{client drift} as one of the key weaknesses that arise when vanilla federated averaging is applied in such a system, especially since each client can independently have different order of tasks. We outline a framework for performing Federated Continual Learning (FCL) by using NetTailor as a candidate continual learning approach and show the extent of the problem of client drift. We show that adaptive federated optimization can reduce the adverse impact of client drift and showcase its effectiveness on CIFAR100, MiniImagenet, and Decathlon benchmarks. Further, we provide an empirical analysis highlighting the interplay between different hyperparameters such as client and server learning rates, the number of local training iterations, and communication rounds. Finally, we evaluate our framework on useful characteristics of federated learning systems such as scalability, robustness to the skewness in clients' data distribution, and stragglers.

Related papers

• Efficient Federated Learning against Heterogeneous and Non-stationary Client Unavailability [23.466997173249034]
  FedAPM includes novel structures that (i) for missed computations due to unavailability with only $(1)O$ additional memory computation with respect to standard FedAvg. We show that FedAPM converges to a stationary point even non-stationary algorithm despite being non-stationary dynamics.
  arXiv  Detail & Related papers  (2024-09-26T00:38:18Z)
• FedLALR: Client-Specific Adaptive Learning Rates Achieve Linear Speedup for Non-IID Data [54.81695390763957]
  Federated learning is an emerging distributed machine learning method. We propose a heterogeneous local variant of AMSGrad, named FedLALR, in which each client adjusts its learning rate. We show that our client-specified auto-tuned learning rate scheduling can converge and achieve linear speedup with respect to the number of clients.
  arXiv  Detail & Related papers  (2023-09-18T12:35:05Z)
• When to Trust Aggregated Gradients: Addressing Negative Client Sampling in Federated Learning [41.51682329500003]
  We propose a novel learning rate adaptation mechanism to adjust the server learning rate for the aggregated gradient in each round. We make theoretical deductions to find a meaningful and robust indicator that is positively related to the optimal server learning rate.
  arXiv  Detail & Related papers  (2023-01-25T03:52:45Z)
• When Do Curricula Work in Federated Learning? [56.88941905240137]
  We find that curriculum learning largely alleviates non-IIDness. The more disparate the data distributions across clients the more they benefit from learning. We propose a novel client selection technique that benefits from the real-world disparity in the clients.
  arXiv  Detail & Related papers  (2022-12-24T11:02:35Z)
• Straggler-Resilient Personalized Federated Learning [55.54344312542944]
  Federated learning allows training models from samples distributed across a large network of clients while respecting privacy and communication restrictions. We develop a novel algorithmic procedure with theoretical speedup guarantees that simultaneously handles two of these hurdles. Our method relies on ideas from representation learning theory to find a global common representation using all clients' data and learn a user-specific set of parameters leading to a personalized solution for each client.
  arXiv  Detail & Related papers  (2022-06-05T01:14:46Z)
• A Fair Federated Learning Framework With Reinforcement Learning [23.675056844328]
  Federated learning (FL) is a paradigm where many clients collaboratively train a model under the coordination of a central server. We propose a reinforcement learning framework, called PG-FFL, which automatically learns a policy to assign aggregation weights to clients. We conduct extensive experiments over diverse datasets to verify the effectiveness of our framework.
  arXiv  Detail & Related papers  (2022-05-26T15:10:16Z)
• Exploiting Shared Representations for Personalized Federated Learning [54.65133770989836]
  We propose a novel federated learning framework and algorithm for learning a shared data representation across clients and unique local heads for each client. Our algorithm harnesses the distributed computational power across clients to perform many local-updates with respect to the low-dimensional local parameters for every update of the representation. This result is of interest beyond federated learning to a broad class of problems in which we aim to learn a shared low-dimensional representation among data distributions.
  arXiv  Detail & Related papers  (2021-02-14T05:36:25Z)
• Straggler-Resilient Federated Learning: Leveraging the Interplay Between Statistical Accuracy and System Heterogeneity [57.275753974812666]
  Federated learning involves learning from data samples distributed across a network of clients while the data remains local. In this paper, we propose a novel straggler-resilient federated learning method that incorporates statistical characteristics of the clients' data to adaptively select the clients in order to speed up the learning procedure.
  arXiv  Detail & Related papers  (2020-12-28T19:21:14Z)
• Federated Continual Learning with Weighted Inter-client Transfer [79.93004004545736]
  We propose a novel federated continual learning framework, Federated Weighted Inter-client Transfer (FedWeIT) FedWeIT decomposes the network weights into global federated parameters and sparse task-specific parameters, and each client receives selective knowledge from other clients. We validate our FedWeIT against existing federated learning and continual learning methods, and our model significantly outperforms them with a large reduction in the communication cost.
  arXiv  Detail & Related papers  (2020-03-06T13:33:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.