Unlocking FedNL: Self-Contained Compute-Optimized Implementation

• URL: http://arxiv.org/abs/2410.08760v2
• Date: Thu, 12 Dec 2024 14:43:48 GMT
• Title: Unlocking FedNL: Self-Contained Compute-Optimized Implementation
• Authors: Konstantin Burlachenko, Peter Richtárik,
• Abstract summary: Federated Learning (FL) is an emerging paradigm that enables intelligent agents to collaboratively train Machine Learning (ML) models in a distributed manner. Recent work introduces a family of Federated Newton Learn (FedNL) algorithms, marking a significant step towards applying second-order methods to FL and large-scale optimization. We present a self-contained implementation of FedNL, FedNL-LS, FedNL-PP for single-node and multi-node settings.
• Score: 56.16884466478886
• License:
• Abstract: Federated Learning (FL) is an emerging paradigm that enables intelligent agents to collaboratively train Machine Learning (ML) models in a distributed manner, eliminating the need for sharing their local data. The recent work (arXiv:2106.02969) introduces a family of Federated Newton Learn (FedNL) algorithms, marking a significant step towards applying second-order methods to FL and large-scale optimization. However, the reference FedNL prototype exhibits three serious practical drawbacks: (i) It requires 4.8 hours to launch a single experiment in a sever-grade workstation; (ii) The prototype only simulates multi-node setting; (iii) Prototype integration into resource-constrained applications is challenging. To bridge the gap between theory and practice, we present a self-contained implementation of FedNL, FedNL-LS, FedNL-PP for single-node and multi-node settings. Our work resolves the aforementioned issues and reduces the wall clock time by x1000. With this FedNL outperforms alternatives for training logistic regression in a single-node -- CVXPY (arXiv:1603.00943), and in a multi-node -- Apache Spark (arXiv:1505.06807), Ray/Scikit-Learn (arXiv:1712.05889). Finally, we propose two practical-orientated compressors for FedNL - adaptive TopLEK and cache-aware RandSeqK, which fulfill the theory of FedNL.

Related papers

• Federated Split Learning with Model Pruning and Gradient Quantization in Wireless Networks [7.439160287320074]
  Federated split learning (FedSL) implements collaborative training across the edge devices and the server through model splitting. We propose a lightweight FedSL scheme, that further alleviates the training burden on resource-constrained edge devices. We conduct theoretical analysis to quantify the convergence performance of the proposed scheme.
  arXiv  Detail & Related papers  (2024-12-09T11:43:03Z)
• Enhancing Spectrum Efficiency in 6G Satellite Networks: A GAIL-Powered Policy Learning via Asynchronous Federated Inverse Reinforcement Learning [67.95280175998792]
  A novel adversarial imitation learning (GAIL)-powered policy learning approach is proposed for optimizing beamforming, spectrum allocation, and remote user equipment (RUE) association ins. We employ inverse RL (IRL) to automatically learn reward functions without manual tuning. We show that the proposed MA-AL method outperforms traditional RL approaches, achieving a $14.6%$ improvement in convergence and reward value.
  arXiv  Detail & Related papers  (2024-09-27T13:05:02Z)
• FedLPS: Heterogeneous Federated Learning for Multiple Tasks with Local Parameter Sharing [14.938531944702193]
  We propose Federated Learning with Local Heterogeneous Sharing (FedLPS) FedLPS uses transfer learning to facilitate the deployment of multiple tasks on a single device by dividing the local model into a shareable encoder and task-specific encoders. FedLPS significantly outperforms the state-of-the-art (SOTA) FL frameworks by up to 4.88% and reduces the computational resource consumption by 21.3%.
  arXiv  Detail & Related papers  (2024-02-13T16:30:30Z)
• pFedLoRA: Model-Heterogeneous Personalized Federated Learning with LoRA Tuning [35.59830784463706]
  Federated learning (FL) is an emerging machine learning paradigm in which a central server coordinates multiple participants (clients) collaboratively to train on decentralized data. We propose a novel and efficient model-heterogeneous personalized Federated learning framework based on LoRA tuning (pFedLoRA) Experiments on two benchmark datasets demonstrate that pFedLoRA outperforms six state-of-the-art baselines.
  arXiv  Detail & Related papers  (2023-10-20T05:24:28Z)
• FedLALR: Client-Specific Adaptive Learning Rates Achieve Linear Speedup for Non-IID Data [54.81695390763957]
  Federated learning is an emerging distributed machine learning method. We propose a heterogeneous local variant of AMSGrad, named FedLALR, in which each client adjusts its learning rate. We show that our client-specified auto-tuned learning rate scheduling can converge and achieve linear speedup with respect to the number of clients.
  arXiv  Detail & Related papers  (2023-09-18T12:35:05Z)
• Parallel Successive Learning for Dynamic Distributed Model Training over Heterogeneous Wireless Networks [50.68446003616802]
  Federated learning (FedL) has emerged as a popular technique for distributing model training over a set of wireless devices. We develop parallel successive learning (PSL), which expands the FedL architecture along three dimensions. Our analysis sheds light on the notion of cold vs. warmed up models, and model inertia in distributed machine learning.
  arXiv  Detail & Related papers  (2022-02-07T05:11:01Z)
• FedNL: Making Newton-Type Methods Applicable to Federated Learning [5.400491728405083]
  We propose a family of Federated Newton Learn (FedNL) methods. FedNL employs a different Hessian learning technique which i) enhances privacy as it does not rely on the training data to be revealed to the coordinating server. We prove local convergence rates that are independent of the condition number, the number of training data points, and compression variance.
  arXiv  Detail & Related papers  (2021-06-05T21:30:11Z)
• Secure Bilevel Asynchronous Vertical Federated Learning with Backward Updating [159.48259714642447]
  Vertical scalable learning (VFL) attracts increasing attention due to the demands of multi-party collaborative modeling and concerns of privacy leakage. We propose a novel bftextlevel parallel architecture (VF$bfB2$), under which three new algorithms, including VF$B2$, are proposed.
  arXiv  Detail & Related papers  (2021-03-01T12:34:53Z)
• FedDANE: A Federated Newton-Type Method [49.9423212899788]
  Federated learning aims to jointly learn low statistical models over massively distributed datasets. We propose FedDANE, an optimization that we adapt from DANE, to handle federated learning.
  arXiv  Detail & Related papers  (2020-01-07T07:44:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.