Second-Order Guarantees in Federated Learning

• URL: http://arxiv.org/abs/2012.01474v1
• Date: Wed, 2 Dec 2020 19:30:08 GMT
• Title: Second-Order Guarantees in Federated Learning
• Authors: Stefan Vlaski, Elsa Rizk, Ali H. Sayed
• Abstract summary: Federated learning is a useful centralized learning from distributed data. This paper focuses on the second-order optimality algorithms in centralized and decentralized settings.
• Score: 49.17137296715029
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Federated learning is a useful framework for centralized learning from distributed data under practical considerations of heterogeneity, asynchrony, and privacy. Federated architectures are frequently deployed in deep learning settings, which generally give rise to non-convex optimization problems. Nevertheless, most existing analysis are either limited to convex loss functions, or only establish first-order stationarity, despite the fact that saddle-points, which are first-order stationary, are known to pose bottlenecks in deep learning. We draw on recent results on the second-order optimality of stochastic gradient algorithms in centralized and decentralized settings, and establish second-order guarantees for a class of federated learning algorithms.

Related papers

• Boosting the Performance of Decentralized Federated Learning via Catalyst Acceleration [66.43954501171292]
  We introduce Catalyst Acceleration and propose an acceleration Decentralized Federated Learning algorithm called DFedCata. DFedCata consists of two main components: the Moreau envelope function, which addresses parameter inconsistencies, and Nesterov's extrapolation step, which accelerates the aggregation phase. Empirically, we demonstrate the advantages of the proposed algorithm in both convergence speed and generalization performance on CIFAR10/100 with various non-iid data distributions.
  arXiv  Detail & Related papers  (2024-10-09T06:17:16Z)
• Fed-Sophia: A Communication-Efficient Second-Order Federated Learning Algorithm [28.505671833986067]
  Federated learning is a machine learning approach where multiple devices collaboratively learn with the help of a parameter server by sharing only their local updates. While gradient-based optimization techniques are widely adopted in this domain, the curvature information that second-order methods exhibit is crucial to guide and speed up the convergence. This paper introduces a scalable second-order method, allowing the adoption of curvature information in federated large models.
  arXiv  Detail & Related papers  (2024-06-10T09:57:30Z)
• Escaping Saddle Points in Heterogeneous Federated Learning via Distributed SGD with Communication Compression [42.85797250438604]
  We propose a novel algorithm Power-EF that only communicates compressed information via a novel error-feedback scheme. Power-EF is the first distributed and compressed SGD algorithm that provably escapes saddle points in heterogeneous FL without any data homogeneity assumptions. Our theory improves/recovers previous results, while extending to much more tolerant settings on the local data.
  arXiv  Detail & Related papers  (2023-10-29T16:24:53Z)
• Stochastic Unrolled Federated Learning [85.6993263983062]
  We introduce UnRolled Federated learning (SURF), a method that expands algorithm unrolling to federated learning. Our proposed method tackles two challenges of this expansion, namely the need to feed whole datasets to the unrolleds and the decentralized nature of federated learning.
  arXiv  Detail & Related papers  (2023-05-24T17:26:22Z)
• Semi-supervised Domain Adaptive Structure Learning [72.01544419893628]
  Semi-supervised domain adaptation (SSDA) is a challenging problem requiring methods to overcome both 1) overfitting towards poorly annotated data and 2) distribution shift across domains. We introduce an adaptive structure learning method to regularize the cooperation of SSL and DA.
  arXiv  Detail & Related papers  (2021-12-12T06:11:16Z)
• FedSKETCH: Communication-Efficient and Private Federated Learning via Sketching [33.54413645276686]
  Communication complexity and privacy are the two key challenges in Federated Learning. We introduce FedSKETCH and FedSKETCHGATE algorithms to address both challenges in Federated learning jointly.
  arXiv  Detail & Related papers  (2020-08-11T19:22:48Z)
• Learning while Respecting Privacy and Robustness to Distributional Uncertainties and Adversarial Data [66.78671826743884]
  The distributionally robust optimization framework is considered for training a parametric model. The objective is to endow the trained model with robustness against adversarially manipulated input data. Proposed algorithms offer robustness with little overhead.
  arXiv  Detail & Related papers  (2020-07-07T18:25:25Z)
• Second-Order Guarantees in Centralized, Federated and Decentralized Nonconvex Optimization [64.26238893241322]
  Simple algorithms have been shown to lead to good empirical results in many contexts. Several works have pursued rigorous analytical justification for studying non optimization problems. A key insight in these analyses is that perturbations play a critical role in allowing local descent algorithms.
  arXiv  Detail & Related papers  (2020-03-31T16:54:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.