FedNL: Making Newton-Type Methods Applicable to Federated Learning
- URL: http://arxiv.org/abs/2106.02969v1
- Date: Sat, 5 Jun 2021 21:30:11 GMT
- Title: FedNL: Making Newton-Type Methods Applicable to Federated Learning
- Authors: Mher Safaryan and Rustem Islamov and Xun Qian and Peter Richt\'arik
- Abstract summary: We propose a family of Federated Newton Learn (FedNL) methods.
FedNL employs a different Hessian learning technique which i) enhances privacy as it does not rely on the training data to be revealed to the coordinating server.
We prove local convergence rates that are independent of the condition number, the number of training data points, and compression variance.
- Score: 5.400491728405083
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Inspired by recent work of Islamov et al (2021), we propose a family of
Federated Newton Learn (FedNL) methods, which we believe is a marked step in
the direction of making second-order methods applicable to FL. In contrast to
the aforementioned work, FedNL employs a different Hessian learning technique
which i) enhances privacy as it does not rely on the training data to be
revealed to the coordinating server, ii) makes it applicable beyond generalized
linear models, and iii) provably works with general contractive compression
operators for compressing the local Hessians, such as Top-$K$ or Rank-$R$,
which are vastly superior in practice. Notably, we do not need to rely on error
feedback for our methods to work with contractive compressors. Moreover, we
develop FedNL-PP, FedNL-CR and FedNL-LS, which are variants of FedNL that
support partial participation, and globalization via cubic regularization and
line search, respectively, and FedNL-BC, which is a variant that can further
benefit from bidirectional compression of gradients and models, i.e., smart
uplink gradient and smart downlink model compression. We prove local
convergence rates that are independent of the condition number, the number of
training data points, and compression variance. Our communication efficient
Hessian learning technique provably learns the Hessian at the optimum. Finally,
we perform a variety of numerical experiments that show that our FedNL methods
have state-of-the-art communication complexity when compared to key baselines.
Related papers
- Unlocking FedNL: Self-Contained Compute-Optimized Implementation [56.16884466478886]
Federated Learning (FL) is an emerging paradigm that enables intelligent agents to collaboratively train Machine Learning (ML) models in a distributed manner.
Recent work introduces a family of Federated Newton Learn (FedNL) algorithms, marking a significant step towards applying second-order methods to FL and large-scale optimization.
We present a self-contained implementation of FedNL, FedNL-LS, FedNL-PP for single-node and multi-node settings.
arXiv Detail & Related papers (2024-10-11T12:19:18Z) - Federated Learning under Partially Class-Disjoint Data via Manifold Reshaping [64.58402571292723]
We propose a manifold reshaping approach called FedMR to calibrate the feature space of local training.
We conduct extensive experiments on a range of datasets to demonstrate that our FedMR achieves much higher accuracy and better communication efficiency.
arXiv Detail & Related papers (2024-05-29T10:56:13Z) - FedLALR: Client-Specific Adaptive Learning Rates Achieve Linear Speedup
for Non-IID Data [54.81695390763957]
Federated learning is an emerging distributed machine learning method.
We propose a heterogeneous local variant of AMSGrad, named FedLALR, in which each client adjusts its learning rate.
We show that our client-specified auto-tuned learning rate scheduling can converge and achieve linear speedup with respect to the number of clients.
arXiv Detail & Related papers (2023-09-18T12:35:05Z) - FedIN: Federated Intermediate Layers Learning for Model Heterogeneity [7.781409257429762]
Federated learning (FL) facilitates edge devices to cooperatively train a global shared model while maintaining the training data locally and privately.
In this study, we propose an FL method called Federated Intermediate Layers Learning (FedIN), supporting heterogeneous models without relying on any public dataset.
Experiment results demonstrate the superior performance of FedIN in heterogeneous model environments compared to state-of-the-art algorithms.
arXiv Detail & Related papers (2023-04-03T07:20:43Z) - FedNew: A Communication-Efficient and Privacy-Preserving Newton-Type
Method for Federated Learning [75.46959684676371]
We introduce a novel framework called FedNew in which there is no need to transmit Hessian information from clients to PS.
FedNew hides the gradient information and results in a privacy-preserving approach compared to the existing state-of-the-art.
arXiv Detail & Related papers (2022-06-17T15:21:39Z) - Acceleration of Federated Learning with Alleviated Forgetting in Local
Training [61.231021417674235]
Federated learning (FL) enables distributed optimization of machine learning models while protecting privacy.
We propose FedReg, an algorithm to accelerate FL with alleviated knowledge forgetting in the local training stage.
Our experiments demonstrate that FedReg not only significantly improves the convergence rate of FL, especially when the neural network architecture is deep.
arXiv Detail & Related papers (2022-03-05T02:31:32Z) - Communication-Efficient Federated Learning with Dual-Side Low-Rank
Compression [8.353152693578151]
Federated learning (FL) is a promising and powerful approach for training deep learning models without sharing the raw data of clients.
We propose a new training method, referred to as federated learning with dual-side low-rank compression (FedDLR)
We show that FedDLR outperforms the state-of-the-art solutions in terms of both the communication and efficiency.
arXiv Detail & Related papers (2021-04-26T09:13:31Z) - Toward Understanding the Influence of Individual Clients in Federated
Learning [52.07734799278535]
Federated learning allows clients to jointly train a global model without sending their private data to a central server.
We defined a new notion called em-Influence, quantify this influence over parameters, and proposed an effective efficient model to estimate this metric.
arXiv Detail & Related papers (2020-12-20T14:34:36Z) - Improving Federated Relational Data Modeling via Basis Alignment and
Weight Penalty [18.096788806121754]
Federated learning (FL) has attracted increasing attention in recent years.
We present a modified version of the graph neural network algorithm that performs federated modeling over Knowledge Graph (KG)
We propose a novel optimization algorithm, named FedAlign, with 1) optimal transportation (OT) for on-client personalization and 2) weight constraint to speed up the convergence.
Empirical results show that our proposed method outperforms the state-of-the-art FL methods, such as FedAVG and FedProx, with better convergence.
arXiv Detail & Related papers (2020-11-23T12:52:18Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.