Personalized Federated Learning with Mixture of Models for Adaptive Prediction and Model Fine-Tuning

• URL: http://arxiv.org/abs/2410.21547v1
• Date: Mon, 28 Oct 2024 21:20:51 GMT
• Title: Personalized Federated Learning with Mixture of Models for Adaptive Prediction and Model Fine-Tuning
• Authors: Pouya M. Ghari, Yanning Shen,
• Abstract summary: This paper develops a novel personalized federated learning algorithm. Each client constructs a personalized model by combining a locally fine-tuned model with multiple federated models. Theoretical analysis and experiments on real datasets corroborate the effectiveness of this approach.
• Score: 22.705411388403036
• License:
• Abstract: Federated learning is renowned for its efficacy in distributed model training, ensuring that users, called clients, retain data privacy by not disclosing their data to the central server that orchestrates collaborations. Most previous work on federated learning assumes that clients possess static batches of training data. However, clients may also need to make real-time predictions on streaming data in non-stationary environments. In such dynamic environments, employing pre-trained models may be inefficient, as they struggle to adapt to the constantly evolving data streams. To address this challenge, clients can fine-tune models online, leveraging their observed data to enhance performance. Despite the potential benefits of client participation in federated online model fine-tuning, existing analyses have not conclusively demonstrated its superiority over local model fine-tuning. To bridge this gap, the present paper develops a novel personalized federated learning algorithm, wherein each client constructs a personalized model by combining a locally fine-tuned model with multiple federated models learned by the server over time. Theoretical analysis and experiments on real datasets corroborate the effectiveness of this approach for real-time predictions and federated model fine-tuning.

Related papers

• Federated Learning with Projected Trajectory Regularization [65.6266768678291]
  Federated learning enables joint training of machine learning models from distributed clients without sharing their local data. One key challenge in federated learning is to handle non-identically distributed data across the clients. We propose a novel federated learning framework with projected trajectory regularization (FedPTR) for tackling the data issue.
  arXiv  Detail & Related papers  (2023-12-22T02:12:08Z)
• Dirichlet-based Uncertainty Quantification for Personalized Federated Learning with Improved Posterior Networks [9.54563359677778]
  This paper presents a new approach to federated learning that allows selecting a model from global and personalized ones. It is achieved through a careful modeling of predictive uncertainties that helps to detect local and global in- and out-of-distribution data. The comprehensive experimental evaluation on the popular real-world image datasets shows the superior performance of the model in the presence of out-of-distribution data.
  arXiv  Detail & Related papers  (2023-12-18T14:30:05Z)
• FAM: fast adaptive federated meta-learning [10.980548731600116]
  We propose a fast adaptive federated meta-learning (FAM) framework for collaboratively learning a single global model. A skeleton network is grown on each client to train a personalized model by learning additional client-specific parameters from local data. The personalized client models outperformed the locally trained models, demonstrating the efficacy of the FAM mechanism.
  arXiv  Detail & Related papers  (2023-08-26T22:54:45Z)
• Efficient Personalized Federated Learning via Sparse Model-Adaptation [47.088124462925684]
  Federated Learning (FL) aims to train machine learning models for multiple clients without sharing their own private data. We propose pFedGate for efficient personalized FL by adaptively and efficiently learning sparse local models. We show that pFedGate achieves superior global accuracy, individual accuracy and efficiency simultaneously over state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-05-04T12:21:34Z)
• Personalizing Federated Learning with Over-the-Air Computations [84.8089761800994]
  Federated edge learning is a promising technology to deploy intelligence at the edge of wireless networks in a privacy-preserving manner. Under such a setting, multiple clients collaboratively train a global generic model under the coordination of an edge server. This paper presents a distributed training paradigm that employs analog over-the-air computation to address the communication bottleneck.
  arXiv  Detail & Related papers  (2023-02-24T08:41:19Z)
• Dataless Knowledge Fusion by Merging Weights of Language Models [51.8162883997512]
  Fine-tuning pre-trained language models has become the prevalent paradigm for building downstream NLP models. This creates a barrier to fusing knowledge across individual models to yield a better single model. We propose a dataless knowledge fusion method that merges models in their parameter space.
  arXiv  Detail & Related papers  (2022-12-19T20:46:43Z)
• Personalized Federated Learning through Local Memorization [10.925242558525683]
  Federated learning allows clients to collaboratively learn statistical models while keeping their data local. Recent personalized federated learning methods train a separate model for each client while still leveraging the knowledge available at other clients. We show on a suite of federated datasets that this approach achieves significantly higher accuracy and fairness than state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-11-17T19:40:07Z)
• Towards Fair Federated Learning with Zero-Shot Data Augmentation [123.37082242750866]
  Federated learning has emerged as an important distributed learning paradigm, where a server aggregates a global model from many client-trained models while having no access to the client data. We propose a novel federated learning system that employs zero-shot data augmentation on under-represented data to mitigate statistical heterogeneity and encourage more uniform accuracy performance across clients in federated networks. We study two variants of this scheme, Fed-ZDAC (federated learning with zero-shot data augmentation at the clients) and Fed-ZDAS (federated learning with zero-shot data augmentation at the server).
  arXiv  Detail & Related papers  (2021-04-27T18:23:54Z)
• Toward Understanding the Influence of Individual Clients in Federated Learning [52.07734799278535]
  Federated learning allows clients to jointly train a global model without sending their private data to a central server. We defined a new notion called em-Influence, quantify this influence over parameters, and proposed an effective efficient model to estimate this metric.
  arXiv  Detail & Related papers  (2020-12-20T14:34:36Z)
• Federated Residual Learning [53.77128418049985]
  We study a new form of federated learning where the clients train personalized local models and make predictions jointly with the server-side shared model. Using this new federated learning framework, the complexity of the central shared model can be minimized while still gaining all the performance benefits that joint training provides.
  arXiv  Detail & Related papers  (2020-03-28T19:55:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.