pFedSOP : Accelerating Training Of Personalized Federated Learning Using Second-Order Optimization

• URL: http://arxiv.org/abs/2506.07159v1
• Date: Sun, 08 Jun 2025 14:09:47 GMT
• Title: pFedSOP : Accelerating Training Of Personalized Federated Learning Using Second-Order Optimization
• Authors: Mrinmay Sen, Chalavadi Krishna Mohan,
• Abstract summary: We propose pFedSOP, which efficiently utilizes second-order optimization in PFL to accelerate the training of personalized models.<n>Our approach first computes a personalized local gradient update using the Gompertz function-based normalized angle between local and global gradient updates.<n>We then use a regularized Fisher Information Matrix (FIM), computed from this personalized update, as an approximation of the Hessian to update the personalized models.
• Score: 4.189643331553922
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Personalized Federated Learning (PFL) enables clients to collaboratively train personalized models tailored to their individual objectives, addressing the challenge of model generalization in traditional Federated Learning (FL) due to high data heterogeneity. However, existing PFL methods often require increased communication rounds to achieve the desired performance, primarily due to slow training caused by the use of first-order optimization, which has linear convergence. Additionally, many of these methods increase local computation because of the additional data fed into the model during the search for personalized local models. One promising solution to this slow training is second-order optimization, known for its quadratic convergence. However, employing it in PFL is challenging due to the Hessian matrix and its inverse. In this paper, we propose pFedSOP, which efficiently utilizes second-order optimization in PFL to accelerate the training of personalized models and enhance performance with fewer communication rounds. Our approach first computes a personalized local gradient update using the Gompertz function-based normalized angle between local and global gradient updates, incorporating client-specific global information. We then use a regularized Fisher Information Matrix (FIM), computed from this personalized gradient update, as an approximation of the Hessian to update the personalized models. This FIM-based second-order optimization speeds up training with fewer communication rounds by tackling the challenges with exact Hessian and avoids additional data being fed into the model during the search for personalized local models. Extensive experiments on heterogeneously partitioned image classification datasets with partial client participation demonstrate that pFedSOP outperforms state-of-the-art FL and PFL algorithms.

Related papers

• FedAPA: Server-side Gradient-Based Adaptive Personalized Aggregation for Federated Learning on Heterogeneous Data [5.906966694759679]
  FedAPA is a novel PFL method featuring a server-side, gradient-based adaptive aggregation strategy to generate personalized models.<n>FedAPA guarantees theoretical convergence and achieves superior accuracy and computational efficiency compared to 10 PFL competitors across three datasets.
  arXiv  Detail & Related papers  (2025-02-11T11:00:58Z)
• Look Back for More: Harnessing Historical Sequential Updates for Personalized Federated Adapter Tuning [50.45027483522507]
  Existing personalized federated learning (PFL) approaches rely solely on the clients' latest updated models.<n>We propose pFedSeq, designed for personalizing adapters to fine-tune a foundation model in FL.<n>In pFedSeq, the server maintains and trains a sequential learner, which processes a sequence of past adapter updates from clients.<n>To effectively capture the cross-client and cross-step relations hidden in previous updates, pFedSeq adopts the powerful selective state space model.
  arXiv  Detail & Related papers  (2025-01-03T06:10:09Z)
• HierSFL: Local Differential Privacy-aided Split Federated Learning in Mobile Edge Computing [7.180235086275924]
  Federated Learning is a promising approach for learning from user data while preserving data privacy. Split Federated Learning is utilized, where clients upload their intermediate model training outcomes to a cloud server for collaborative server-client model training. This methodology facilitates resource-constrained clients' participation in model training but also increases the training time and communication overhead. We propose a novel algorithm, called Hierarchical Split Federated Learning (HierSFL), that amalgamates models at the edge and cloud phases.
  arXiv  Detail & Related papers  (2024-01-16T09:34:10Z)
• ZooPFL: Exploring Black-box Foundation Models for Personalized Federated Learning [95.64041188351393]
  This paper endeavors to solve both the challenges of limited resources and personalization. We propose a method named ZOOPFL that uses Zeroth-Order Optimization for Personalized Federated Learning. To reduce the computation costs and enhance personalization, we propose input surgery to incorporate an auto-encoder with low-dimensional and client-specific embeddings.
  arXiv  Detail & Related papers  (2023-10-08T12:26:13Z)
• Personalized Federated Learning under Mixture of Distributions [98.25444470990107]
  We propose a novel approach to Personalized Federated Learning (PFL), which utilizes Gaussian mixture models (GMM) to fit the input data distributions across diverse clients. FedGMM possesses an additional advantage of adapting to new clients with minimal overhead, and it also enables uncertainty quantification. Empirical evaluations on synthetic and benchmark datasets demonstrate the superior performance of our method in both PFL classification and novel sample detection.
  arXiv  Detail & Related papers  (2023-05-01T20:04:46Z)
• Visual Prompt Based Personalized Federated Learning [83.04104655903846]
  We propose a novel PFL framework for image classification tasks, dubbed pFedPT, that leverages personalized visual prompts to implicitly represent local data distribution information of clients. Experiments on the CIFAR10 and CIFAR100 datasets show that pFedPT outperforms several state-of-the-art (SOTA) PFL algorithms by a large margin in various settings.
  arXiv  Detail & Related papers  (2023-03-15T15:02:15Z)
• Acceleration of Federated Learning with Alleviated Forgetting in Local Training [61.231021417674235]
  Federated learning (FL) enables distributed optimization of machine learning models while protecting privacy. We propose FedReg, an algorithm to accelerate FL with alleviated knowledge forgetting in the local training stage. Our experiments demonstrate that FedReg not only significantly improves the convergence rate of FL, especially when the neural network architecture is deep.
  arXiv  Detail & Related papers  (2022-03-05T02:31:32Z)
• Achieving Personalized Federated Learning with Sparse Local Models [75.76854544460981]
  Federated learning (FL) is vulnerable to heterogeneously distributed data. To counter this issue, personalized FL (PFL) was proposed to produce dedicated local models for each individual user. Existing PFL solutions either demonstrate unsatisfactory generalization towards different model architectures or cost enormous extra computation and memory. We proposeFedSpa, a novel PFL scheme that employs personalized sparse masks to customize sparse local models on the edge.
  arXiv  Detail & Related papers  (2022-01-27T08:43:11Z)
• Parameterized Knowledge Transfer for Personalized Federated Learning [11.223753730705374]
  We propose a novel training framework to employ personalized models for different clients. It is demonstrated that the proposed framework is the first federated learning paradigm that realizes personalized model training.
  arXiv  Detail & Related papers  (2021-11-04T13:41:45Z)
• Personalized Federated Learning with Moreau Envelopes [16.25105865597947]
  Federated learning (FL) is a decentralized and privacy-preserving machine learning technique. One challenge associated with FL is statistical diversity among clients. We propose an algorithm for personalized FL (FedFedMe) using envelopes regularized loss function.
  arXiv  Detail & Related papers  (2020-06-16T00:55:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.