On the Convergence of Continual Federated Learning Using Incrementally Aggregated Gradients

• URL: http://arxiv.org/abs/2411.07959v1
• Date: Tue, 12 Nov 2024 17:36:20 GMT
• Title: On the Convergence of Continual Federated Learning Using Incrementally Aggregated Gradients
• Authors: Satish Kumar Keshri, Nazreen Shah, Ranjitha Prasad,
• Abstract summary: The holy grail of machine learning is to enable Continual Federated Learning (CFL) to enhance the efficiency, privacy, and scalability of AI systems while learning from streaming data. We propose a novel replay-memory based federated strategy consisting of edge-based gradient updates on memory and aggregated gradients on the current data. We empirically show that C-FLAG outperforms several state-of-the-art baselines on both task and class-incremental settings with respect to metrics such as accuracy and forgetting.
• Score: 2.2530496464901106
• License:
• Abstract: The holy grail of machine learning is to enable Continual Federated Learning (CFL) to enhance the efficiency, privacy, and scalability of AI systems while learning from streaming data. The primary challenge of a CFL system is to overcome global catastrophic forgetting, wherein the accuracy of the global model trained on new tasks declines on the old tasks. In this work, we propose Continual Federated Learning with Aggregated Gradients (C-FLAG), a novel replay-memory based federated strategy consisting of edge-based gradient updates on memory and aggregated gradients on the current data. We provide convergence analysis of the C-FLAG approach which addresses forgetting and bias while converging at a rate of $O(1/\sqrt{T})$ over $T$ communication rounds. We formulate an optimization sub-problem that minimizes catastrophic forgetting, translating CFL into an iterative algorithm with adaptive learning rates that ensure seamless learning across tasks. We empirically show that C-FLAG outperforms several state-of-the-art baselines on both task and class-incremental settings with respect to metrics such as accuracy and forgetting.

Related papers

• Buffer-based Gradient Projection for Continual Federated Learning [16.879024856283323]
  Fed-A-GEM mitigates catastrophic forgetting by leveraging local buffer samples and aggregated buffer gradients. Our experiments on standard benchmarks show consistent performance improvements across diverse scenarios.
  arXiv  Detail & Related papers  (2024-09-03T03:50:19Z)
• CG-FedLLM: How to Compress Gradients in Federated Fune-tuning for Large Language Models [21.919883617413358]
  This study introduces an innovative approach to compress gradients to improve communication efficiency during Large-Language Models (LLMs) We also present a series of experimental analyses focusing on the signal-to-noise ratio, compression rate, and robustness within this privacy-centric framework.
  arXiv  Detail & Related papers  (2024-05-22T15:32:38Z)
• Adaptive Coded Federated Learning: Privacy Preservation and Straggler Mitigation [33.56146654796337]
  A coded federated learning framework has been proposed to mitigate the negative impact of stragglers. We propose a new method named adaptive coded federated learning (ACFL) to overcome this drawback.
  arXiv  Detail & Related papers  (2024-03-22T01:51:48Z)
• Communication Efficient ConFederated Learning: An Event-Triggered SAGA Approach [67.27031215756121]
  Federated learning (FL) is a machine learning paradigm that targets model training without gathering the local data over various data sources. Standard FL, which employs a single server, can only support a limited number of users, leading to degraded learning capability. In this work, we consider a multi-server FL framework, referred to as emphConfederated Learning (CFL) in order to accommodate a larger number of users.
  arXiv  Detail & Related papers  (2024-02-28T03:27:10Z)
• FedLALR: Client-Specific Adaptive Learning Rates Achieve Linear Speedup for Non-IID Data [54.81695390763957]
  Federated learning is an emerging distributed machine learning method. We propose a heterogeneous local variant of AMSGrad, named FedLALR, in which each client adjusts its learning rate. We show that our client-specified auto-tuned learning rate scheduling can converge and achieve linear speedup with respect to the number of clients.
  arXiv  Detail & Related papers  (2023-09-18T12:35:05Z)
• FedDA: Faster Framework of Local Adaptive Gradient Methods via Restarted Dual Averaging [104.41634756395545]
  Federated learning (FL) is an emerging learning paradigm to tackle massively distributed data. We propose textbfFedDA, a novel framework for local adaptive gradient methods. We show that textbfFedDA-MVR is the first adaptive FL algorithm that achieves this rate.
  arXiv  Detail & Related papers  (2023-02-13T05:10:30Z)
• Faster Adaptive Federated Learning [84.38913517122619]
  Federated learning has attracted increasing attention with the emergence of distributed data. In this paper, we propose an efficient adaptive algorithm (i.e., FAFED) based on momentum-based variance reduced technique in cross-silo FL.
  arXiv  Detail & Related papers  (2022-12-02T05:07:50Z)
• Stochastic Coded Federated Learning: Theoretical Analysis and Incentive Mechanism Design [18.675244280002428]
  We propose a novel FL framework named coded federated learning (SCFL) that leverages coded computing techniques. In SCFL, each edge device uploads a privacy-preserving coded dataset to the server, which is generated by adding noise to the projected local dataset. We show that SCFL learns a better model within the given time and achieves a better privacy-performance tradeoff than the baseline methods.
  arXiv  Detail & Related papers  (2022-11-08T09:58:36Z)
• Federated Stochastic Gradient Descent Begets Self-Induced Momentum [151.4322255230084]
  Federated learning (FL) is an emerging machine learning method that can be applied in mobile edge systems. We show that running to the gradient descent (SGD) in such a setting can be viewed as adding a momentum-like term to the global aggregation process.
  arXiv  Detail & Related papers  (2022-02-17T02:01:37Z)
• Local Learning Matters: Rethinking Data Heterogeneity in Federated Learning [61.488646649045215]
  Federated learning (FL) is a promising strategy for performing privacy-preserving, distributed learning with a network of clients (i.e., edge devices)
  arXiv  Detail & Related papers  (2021-11-28T19:03:39Z)
• Adaptive Gradient Sparsification for Efficient Federated Learning: An Online Learning Approach [11.986523531539165]
  Federated learning (FL) is an emerging technique for training machine learning models using geographically dispersed data. gradient sparsification (GS) can be applied, where instead of the full gradient, only a small subset of important elements of the gradient is communicated. We propose a novel online learning formulation and algorithm for automatically determining the near-optimal communication and trade-off.
  arXiv  Detail & Related papers  (2020-01-14T13:09:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.