FedPM: Federated Learning Using Second-order Optimization with Preconditioned Mixing of Local Parameters

• URL: http://arxiv.org/abs/2511.09100v1
• Date: Thu, 13 Nov 2025 01:31:46 GMT
• Title: FedPM: Federated Learning Using Second-order Optimization with Preconditioned Mixing of Local Parameters
• Authors: Hiro Ishii, Kenta Niwa, Hiroshi Sawada, Akinori Fujino, Noboru Harada, Rio Yokota,
• Abstract summary: Federated Preconditioned Mixing (FedPM) is a novel Federated Learning (FL) method that leverages second-order optimization.<n>Our FedPM introduces preconditioned mixing of local parameters on the server, effectively mitigating drift in local preconditioners.<n>Results show significant improvements with FedPM in the test accuracy compared to conventional methods incorporating simple mixing.
• Score: 24.28609109854982
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose Federated Preconditioned Mixing (FedPM), a novel Federated Learning (FL) method that leverages second-order optimization. Prior methods--such as LocalNewton, LTDA, and FedSophia--have incorporated second-order optimization in FL by performing iterative local updates on clients and applying simple mixing of local parameters on the server. However, these methods often suffer from drift in local preconditioners, which significantly disrupts the convergence of parameter training, particularly in heterogeneous data settings. To overcome this issue, we refine the update rules by decomposing the ideal second-order update--computed using globally preconditioned global gradients--into parameter mixing on the server and local parameter updates on clients. As a result, our FedPM introduces preconditioned mixing of local parameters on the server, effectively mitigating drift in local preconditioners. We provide a theoretical convergence analysis demonstrating a superlinear rate for strongly convex objectives in scenarios involving a single local update. To demonstrate the practical benefits of FedPM, we conducted extensive experiments. The results showed significant improvements with FedPM in the test accuracy compared to conventional methods incorporating simple mixing, fully leveraging the potential of second-order optimization.

Related papers

• FedMuon: Accelerating Federated Learning with Matrix Orthogonalization [27.47081354557656]
  The core bottleneck of Federated Learning (FL) lies in the communication rounds.<n>Existing FL methods still primarily use element-wise locals (Adam/SGD), neglecting the geometric structure of the weight matrices.<n>We propose a novel Federated Muon (FedMuon), which incorporates two key techniques.
  arXiv  Detail & Related papers  (2025-10-31T11:41:16Z)
• Adaptive Deadline and Batch Layered Synchronized Federated Learning [66.93447103966439]
  Federated learning (FL) enables collaborative model training across distributed edge devices while preserving data privacy, and typically operates in a round-based synchronous manner.<n>We propose ADEL-FL, a novel framework that jointly optimize per-round deadlines and user-specific batch sizes for layer-wise aggregation.
  arXiv  Detail & Related papers  (2025-05-29T19:59:18Z)
• Ferret: Federated Full-Parameter Tuning at Scale for Large Language Models [54.02863371927658]
  Large Language Models (LLMs) have become indispensable in numerous real-world applications.<n>Ferret is the first first-order method with shared randomness to enable scalable full- parameter tuning of LLMs.<n>Ferret achieves high computational efficiency, reduced communication overhead, and fast convergence.
  arXiv  Detail & Related papers  (2024-09-10T07:28:13Z)
• Efficient Federated Learning via Local Adaptive Amended Optimizer with Linear Speedup [90.26270347459915]
  We propose a novel momentum-based algorithm via utilizing the global descent locally adaptive. textitLADA could greatly reduce the communication rounds and achieves higher accuracy than several baselines.
  arXiv  Detail & Related papers  (2023-07-30T14:53:21Z)
• FedSpeed: Larger Local Interval, Less Communication Round, and Higher Generalization Accuracy [84.45004766136663]
  Federated learning is an emerging distributed machine learning framework. It suffers from the non-vanishing biases introduced by the local inconsistent optimal and the rugged client-drifts by the local over-fitting. We propose a novel and practical method, FedSpeed, to alleviate the negative impacts posed by these problems.
  arXiv  Detail & Related papers  (2023-02-21T03:55:29Z)
• On Second-order Optimization Methods for Federated Learning [59.787198516188425]
  We evaluate the performance of several second-order distributed methods with local steps in the federated learning setting. We propose a novel variant that uses second-order local information for updates and a global line search to counteract the resulting local specificity.
  arXiv  Detail & Related papers  (2021-09-06T12:04:08Z)
• Local Adaptivity in Federated Learning: Convergence and Consistency [25.293584783673413]
  Federated learning (FL) framework trains a machine learning model using decentralized data stored at edge client devices by periodically aggregating locally trained models. We show in both theory and practice that while local adaptive methods can accelerate convergence, they can cause a non-vanishing solution bias. We propose correction techniques to overcome this inconsistency and complement the local adaptive methods for FL.
  arXiv  Detail & Related papers  (2021-06-04T07:36:59Z)
• Faster Non-Convex Federated Learning via Global and Local Momentum [57.52663209739171]
  textttFedGLOMO is the first (first-order) FLtexttFedGLOMO algorithm. Our algorithm is provably optimal even with communication between the clients and the server.
  arXiv  Detail & Related papers  (2020-12-07T21:05:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.