FedSelect: Customized Selection of Parameters for Fine-Tuning during Personalized Federated Learning

• URL: http://arxiv.org/abs/2306.13264v4
• Date: Sat, 8 Jun 2024 20:15:48 GMT
• Title: FedSelect: Customized Selection of Parameters for Fine-Tuning during Personalized Federated Learning
• Authors: Rishub Tamirisa, John Won, Chengjun Lu, Ron Arel, Andy Zhou,
• Abstract summary: We propose a novel FL framework, FedSelect, that directly personalizes both client subnetwork structure and parameters. We show that this method achieves promising results on CIFAR-10.
• Score: 0.873811641236639
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advancements in federated learning (FL) seek to increase client-level performance by fine-tuning client parameters on local data or personalizing architectures for the local task. Existing methods for such personalization either prune a global model or fine-tune a global model on a local client distribution. However, these existing methods either personalize at the expense of retaining important global knowledge, or predetermine network layers for fine-tuning, resulting in suboptimal storage of global knowledge within client models. Enlightened by the lottery ticket hypothesis, we first introduce a hypothesis for finding optimal client subnetworks to locally fine-tune while leaving the rest of the parameters frozen. We then propose a novel FL framework, FedSelect, using this procedure that directly personalizes both client subnetwork structure and parameters, via the simultaneous discovery of optimal parameters for personalization and the rest of parameters for global aggregation during training. We show that this method achieves promising results on CIFAR-10.

Related papers

• FedSelect: Personalized Federated Learning with Customized Selection of Parameters for Fine-Tuning [9.22574528776347]
  FedSelect is a novel PFL algorithm inspired by the iterative subnetwork discovery procedure used for the Lottery Ticket Hypothesis. We show that FedSelect outperforms recent state-of-the-art PFL algorithms under challenging client data heterogeneity settings.
  arXiv  Detail & Related papers  (2024-04-03T05:36:21Z)
• Learn What You Need in Personalized Federated Learning [53.83081622573734]
  $textitLearn2pFed$ is a novel algorithm-unrolling-based personalized federated learning framework. We show that $textitLearn2pFed$ significantly outperforms previous personalized federated learning methods.
  arXiv  Detail & Related papers  (2024-01-16T12:45:15Z)
• LEFL: Low Entropy Client Sampling in Federated Learning [6.436397118145477]
  Federated learning (FL) is a machine learning paradigm where multiple clients collaborate to optimize a single global model using their private data. We propose LEFL, an alternative sampling strategy by performing a one-time clustering of clients based on their model's learned high-level features. We show of sampled clients selected with this approach yield a low relative entropy with respect to the global data distribution.
  arXiv  Detail & Related papers  (2023-12-29T01:44:20Z)
• Towards Instance-adaptive Inference for Federated Learning [80.38701896056828]
  Federated learning (FL) is a distributed learning paradigm that enables multiple clients to learn a powerful global model by aggregating local training. In this paper, we present a novel FL algorithm, i.e., FedIns, to handle intra-client data heterogeneity by enabling instance-adaptive inference in the FL framework. Our experiments show that our FedIns outperforms state-of-the-art FL algorithms, e.g., a 6.64% improvement against the top-performing method with less than 15% communication cost on Tiny-ImageNet.
  arXiv  Detail & Related papers  (2023-08-11T09:58:47Z)
• Take Your Pick: Enabling Effective Personalized Federated Learning within Low-dimensional Feature Space [22.433424519577642]
  We propose a novel PFL framework named FedPick. FedPick achieves PFL in the low-dimensional feature space by selecting task-relevant features adaptively for each client.
  arXiv  Detail & Related papers  (2023-07-26T07:07:27Z)
• FedSampling: A Better Sampling Strategy for Federated Learning [81.85411484302952]
  Federated learning (FL) is an important technique for learning models from decentralized data in a privacy-preserving way. Existing FL methods usually uniformly sample clients for local model learning in each round. We propose a novel data uniform sampling strategy for federated learning (FedSampling)
  arXiv  Detail & Related papers  (2023-06-25T13:38:51Z)
• Visual Prompt Based Personalized Federated Learning [83.04104655903846]
  We propose a novel PFL framework for image classification tasks, dubbed pFedPT, that leverages personalized visual prompts to implicitly represent local data distribution information of clients. Experiments on the CIFAR10 and CIFAR100 datasets show that pFedPT outperforms several state-of-the-art (SOTA) PFL algorithms by a large margin in various settings.
  arXiv  Detail & Related papers  (2023-03-15T15:02:15Z)
• Personalized Federated Learning with Exact Stochastic Gradient Descent [15.666401346622575]
  We present a new approach for personalized benchmarks that achieves exact updates of gradientSGD descent. We propose a novel SGD-type scheme where at each optimization round, randomly selected clients perform updates over their client-specific towards optimizing the loss function. This allows to an exact minimization of the personalized parameters are performed by the clients and those of the common ones.
  arXiv  Detail & Related papers  (2022-02-20T16:11:20Z)
• A Bayesian Federated Learning Framework with Online Laplace Approximation [144.7345013348257]
  Federated learning allows multiple clients to collaboratively learn a globally shared model. We propose a novel FL framework that uses online Laplace approximation to approximate posteriors on both the client and server side. We achieve state-of-the-art results on several benchmarks, clearly demonstrating the advantages of the proposed method.
  arXiv  Detail & Related papers  (2021-02-03T08:36:58Z)
• Personalized Federated Learning with First Order Model Optimization [76.81546598985159]
  We propose an alternative to federated learning, where each client federates with other relevant clients to obtain a stronger model per client-specific objectives. We do not assume knowledge of underlying data distributions or client similarities, and allow each client to optimize for arbitrary target distributions of interest. Our method outperforms existing alternatives, while also enabling new features for personalized FL such as transfer outside of local data distributions.
  arXiv  Detail & Related papers  (2020-12-15T19:30:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.