Accelerated Training of Federated Learning via Second-Order Methods

• URL: http://arxiv.org/abs/2505.23588v1
• Date: Thu, 29 May 2025 16:00:34 GMT
• Title: Accelerated Training of Federated Learning via Second-Order Methods
• Authors: Mrinmay Sen, Sidhant R Nair, C Krishna Mohan,
• Abstract summary: This paper explores second-order optimization methods in Federated Learning (FL)<n>It addresses the critical challenges of slow convergence and the excessive communication rounds required to achieve optimal performance from the global model.
• Score: 2.1165011830664673
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper explores second-order optimization methods in Federated Learning (FL), addressing the critical challenges of slow convergence and the excessive communication rounds required to achieve optimal performance from the global model. While existing surveys in FL primarily focus on challenges related to statistical and device label heterogeneity, as well as privacy and security concerns in first-order FL methods, less attention has been given to the issue of slow model training. This slow training often leads to the need for excessive communication rounds or increased communication costs, particularly when data across clients are highly heterogeneous. In this paper, we examine various FL methods that leverage second-order optimization to accelerate the training process. We provide a comprehensive categorization of state-of-the-art second-order FL methods and compare their performance based on convergence speed, computational cost, memory usage, transmission overhead, and generalization of the global model. Our findings show the potential of incorporating Hessian curvature through second-order optimization into FL and highlight key challenges, such as the efficient utilization of Hessian and its inverse in FL. This work lays the groundwork for future research aimed at developing scalable and efficient federated optimization methods for improving the training of the global model in FL.

Related papers

• pFedSOP : Accelerating Training Of Personalized Federated Learning Using Second-Order Optimization [4.189643331553922]
  We propose pFedSOP, which efficiently utilizes second-order optimization in PFL to accelerate the training of personalized models.<n>Our approach first computes a personalized local gradient update using the Gompertz function-based normalized angle between local and global gradient updates.<n>We then use a regularized Fisher Information Matrix (FIM), computed from this personalized update, as an approximation of the Hessian to update the personalized models.
  arXiv  Detail & Related papers  (2025-06-08T14:09:47Z)
• Over-the-Air Fair Federated Learning via Multi-Objective Optimization [52.295563400314094]
  We propose an over-the-air fair federated learning algorithm (OTA-FFL) to train fair FL models.<n>Experiments demonstrate the superiority of OTA-FFL in achieving fairness and robust performance.
  arXiv  Detail & Related papers  (2025-01-06T21:16:51Z)
• Understanding Generalization of Federated Learning: the Trade-off between Model Stability and Optimization [22.577751005038543]
  Federated Learning (FL) is a distributed learning approach that trains machine learning models across multiple devices.<n>FL often faces challenges due to data heterogeneity, leading to inconsistent local optima among clients.<n>This paper introduces an innovative generalization dynamics analysis framework, named as Libra, for algorithm-dependent excess risk minimization.
  arXiv  Detail & Related papers  (2024-11-25T11:43:22Z)
• FADAS: Towards Federated Adaptive Asynchronous Optimization [56.09666452175333]
  Federated learning (FL) has emerged as a widely adopted training paradigm for privacy-preserving machine learning. This paper introduces federated adaptive asynchronous optimization, named FADAS, a novel method that incorporates asynchronous updates into adaptive federated optimization with provable guarantees. We rigorously establish the convergence rate of the proposed algorithms and empirical results demonstrate the superior performance of FADAS over other asynchronous FL baselines.
  arXiv  Detail & Related papers  (2024-07-25T20:02:57Z)
• FedEGG: Federated Learning with Explicit Global Guidance [90.04705121816185]
  Federated Learning (FL) holds great potential for diverse applications owing to its privacy-preserving nature.<n>Existing methods help address these challenges via optimization-based client constraints, adaptive client selection, or the use of pre-trained models or synthetic data.<n>We present bftextFedEGG, a new FL algorithm that constructs a global guiding task using a well-defined, easy-to-converge learning task.
  arXiv  Detail & Related papers  (2024-04-18T04:25:21Z)
• FAGH: Accelerating Federated Learning with Approximated Global Hessian [0.27309692684728615]
  We propose an FL with approximated global Hessian (FAGH) method to accelerate FL training. FAGH accelerates the convergence of global model training, leading to the reduced number of communication rounds and thus the shortened training time. Notably, FAGH outperforms several state-of-the-art FL training methods.
  arXiv  Detail & Related papers  (2024-03-16T23:24:03Z)
• Communication-Efficient Heterogeneous Federated Learning with Generalized Heavy-Ball Momentum [9.205317127088168]
  Federated Learning (FL) has emerged as the state-of-the-art approach for learning from decentralized data in privacy-constrained scenarios. Existing approaches are not sufficiently robust, do not perform well in large-scale scenarios, and are not communication efficient. We propose a novel Generalized Heavy-Ball Momentum (GHBM) as an adaptive, communication-efficient by-design instance of GHBM.
  arXiv  Detail & Related papers  (2023-11-30T14:17:57Z)
• Semi-Federated Learning: Convergence Analysis and Optimization of A Hybrid Learning Framework [70.83511997272457]
  We propose a semi-federated learning (SemiFL) paradigm to leverage both the base station (BS) and devices for a hybrid implementation of centralized learning (CL) and FL. We propose a two-stage algorithm to solve this intractable problem, in which we provide the closed-form solutions to the beamformers.
  arXiv  Detail & Related papers  (2023-10-04T03:32:39Z)
• Faster Adaptive Federated Learning [84.38913517122619]
  Federated learning has attracted increasing attention with the emergence of distributed data. In this paper, we propose an efficient adaptive algorithm (i.e., FAFED) based on momentum-based variance reduced technique in cross-silo FL.
  arXiv  Detail & Related papers  (2022-12-02T05:07:50Z)
• Over-the-Air Federated Learning via Second-Order Optimization [37.594140209854906]
  Federated learning (FL) could result in task-oriented data traffic flows over wireless networks with limited radio resources. We propose a novel over-the-air second-order federated optimization algorithm to simultaneously reduce the communication rounds and enable low-latency global model aggregation.
  arXiv  Detail & Related papers  (2022-03-29T12:39:23Z)
• Accelerating Federated Learning with a Global Biased Optimiser [16.69005478209394]
  Federated Learning (FL) is a recent development in the field of machine learning that collaboratively trains models without the training data leaving client devices. We propose a novel, generalised approach for applying adaptive optimisation techniques to FL with the Federated Global Biased Optimiser (FedGBO) algorithm. FedGBO accelerates FL by applying a set of global biased optimiser values during the local training phase of FL, which helps to reduce client-drift' from non-IID data.
  arXiv  Detail & Related papers  (2021-08-20T12:08:44Z)
• Towards More Efficient Federated Learning with Better Optimization Objects [1.126965032229697]
  Federated Learning (FL) is a privacy-protected machine learning paradigm that allows model to be trained directly at the edge without uploading data. One of the biggest challenges faced by FL in practical applications is the heterogeneity of edge node data, which will slow down the convergence speed and degrade the performance of the model. We propose to use the aggregation of all models obtained in the past as new constraint target to further improve the performance of such algorithms.
  arXiv  Detail & Related papers  (2021-08-19T09:29:17Z)
• Dynamic Attention-based Communication-Efficient Federated Learning [85.18941440826309]
  Federated learning (FL) offers a solution to train a global machine learning model. FL suffers performance degradation when client data distribution is non-IID. We propose a new adaptive training algorithm $textttAdaFL$ to combat this degradation.
  arXiv  Detail & Related papers  (2021-08-12T14:18:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.