Collaborative and Efficient Personalization with Mixtures of Adaptors

• URL: http://arxiv.org/abs/2410.03497v2
• Date: Sat, 08 Mar 2025 19:24:41 GMT
• Title: Collaborative and Efficient Personalization with Mixtures of Adaptors
• Authors: Abdulla Jasem Almansoori, Samuel Horváth, Martin Takáč,
• Abstract summary: Federated Low-Rank Adaptive Learning (FLoRAL) allows clients to personalize in groups by mixing between low-rank adaptors.<n>FLoRAL is a model parameterization that casts personalized federated learning as a multi-task learning problem.
• Score: 5.195669033269619
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Heterogenous data is prevalent in real-world federated learning. We propose a parameter-efficient framework, Federated Low-Rank Adaptive Learning (FLoRAL), that allows clients to personalize in groups by mixing between low-rank adaptors, where the mixtures are client-specific. FLoRAL is a model parameterization that casts personalized federated learning as a multi-task learning problem, with weight sharing as an implicit regularizer. It is memory-efficient, as the personalized parameters (i.e., base model + adaptors) are all federated. Our results show that FLoRAL can generalize better than a mixture of full models when data are scarce. It can also consistently personalize better than models with a locally tuned adaptor per client. This demonstrates the benefits of "federated personalization" and its robustness against overfitting. We derive the convergence rates and show theoretically that FLoRAL can lead to better variance reduction of the base model's gradients.

Related papers

• Client-Centric Federated Adaptive Optimization [78.30827455292827]
  Federated Learning (FL) is a distributed learning paradigm where clients collaboratively train a model while keeping their own data private. We propose Federated-Centric Adaptive Optimization, which is a class of novel federated optimization approaches.
  arXiv  Detail & Related papers  (2025-01-17T04:00:50Z)
• Personalized federated learning based on feature fusion [2.943623084019036]
  Federated learning enables distributed clients to collaborate on training while storing their data locally to protect client privacy. We propose a personalized federated learning approach called pFedPM. In our process, we replace traditional gradient uploading with feature uploading, which helps reduce communication costs and allows for heterogeneous client models.
  arXiv  Detail & Related papers  (2024-06-24T12:16:51Z)
• pFedAFM: Adaptive Feature Mixture for Batch-Level Personalization in Heterogeneous Federated Learning [34.01721941230425]
  We propose a model-heterogeneous personalized Federated learning approach with Adaptive Feature Mixture (pFedAFM) for supervised learning tasks. It significantly outperforms 7 state-of-the-art MHPFL methods, achieving up to 7.93% accuracy improvement.
  arXiv  Detail & Related papers  (2024-04-27T09:52:59Z)
• Loop Improvement: An Efficient Approach for Extracting Shared Features from Heterogeneous Data without Central Server [16.249442761713322]
  "Loop Improvement" (LI) is a novel method enhancing this separation and feature extraction without necessitating a central server or data interchange among participants. In personalized federated learning environments, LI consistently outperforms the advanced FedALA algorithm in accuracy across diverse scenarios. LI's adaptability extends to multi-task learning, streamlining the extraction of common features across tasks and obviating the need for simultaneous training.
  arXiv  Detail & Related papers  (2024-03-21T12:59:24Z)
• CoDream: Exchanging dreams instead of models for federated aggregation with heterogeneous models [8.85591781936764]
  We present a novel framework called CoDream, where clients collaboratively optimize randomly data. Our key insight is that jointly optimizing this data can effectively capture the properties of the global data distribution. We empirically validate CoDream on standard FL tasks, demonstrating competitive performance despite not sharing model parameters.
  arXiv  Detail & Related papers  (2024-02-25T03:07:32Z)
• Learn What You Need in Personalized Federated Learning [53.83081622573734]
  $textitLearn2pFed$ is a novel algorithm-unrolling-based personalized federated learning framework. We show that $textitLearn2pFed$ significantly outperforms previous personalized federated learning methods.
  arXiv  Detail & Related papers  (2024-01-16T12:45:15Z)
• Federated Skewed Label Learning with Logits Fusion [23.062650578266837]
  Federated learning (FL) aims to collaboratively train a shared model across multiple clients without transmitting their local data. We propose FedBalance, which corrects the optimization bias among local models by calibrating their logits. Our method can gain 13% higher average accuracy compared with state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-11-14T14:37:33Z)
• FedLALR: Client-Specific Adaptive Learning Rates Achieve Linear Speedup for Non-IID Data [54.81695390763957]
  Federated learning is an emerging distributed machine learning method. We propose a heterogeneous local variant of AMSGrad, named FedLALR, in which each client adjusts its learning rate. We show that our client-specified auto-tuned learning rate scheduling can converge and achieve linear speedup with respect to the number of clients.
  arXiv  Detail & Related papers  (2023-09-18T12:35:05Z)
• Towards Instance-adaptive Inference for Federated Learning [80.38701896056828]
  Federated learning (FL) is a distributed learning paradigm that enables multiple clients to learn a powerful global model by aggregating local training. In this paper, we present a novel FL algorithm, i.e., FedIns, to handle intra-client data heterogeneity by enabling instance-adaptive inference in the FL framework. Our experiments show that our FedIns outperforms state-of-the-art FL algorithms, e.g., a 6.64% improvement against the top-performing method with less than 15% communication cost on Tiny-ImageNet.
  arXiv  Detail & Related papers  (2023-08-11T09:58:47Z)
• Locally Adaptive Federated Learning [30.19411641685853]
  Federated learning is a paradigm of distributed machine learning in which multiple clients coordinate with a central server to learn a model. Standard federated optimization methods such as Federated Averaging (FedAvg) ensure generalization among the clients. We propose locally federated learning algorithms, that leverage the local geometric information for each client function.
  arXiv  Detail & Related papers  (2023-07-12T17:02:32Z)
• FedJETs: Efficient Just-In-Time Personalization with Federated Mixture of Experts [48.78037006856208]
  FedJETs is a novel solution by using a Mixture-of-Experts (MoE) framework within a Federated Learning (FL) setup. Our method leverages the diversity of the clients to train specialized experts on different subsets of classes, and a gating function to route the input to the most relevant expert(s) Our approach can improve accuracy up to 18% in state of the art FL settings, while maintaining competitive zero-shot performance.
  arXiv  Detail & Related papers  (2023-06-14T15:47:52Z)
• Federated Learning for Semantic Parsing: Task Formulation, Evaluation Setup, New Algorithms [29.636944156801327]
  Multiple clients collaboratively train one global model without sharing their semantic parsing data. Lorar adjusts each client's contribution to the global model update based on its training loss reduction during each round. Clients with smaller datasets enjoy larger performance gains.
  arXiv  Detail & Related papers  (2023-05-26T19:25:49Z)
• Towards More Suitable Personalization in Federated Learning via Decentralized Partial Model Training [67.67045085186797]
  Almost all existing systems have to face large communication burdens if the central FL server fails. It personalizes the "right" in the deep models by alternately updating the shared and personal parameters. To further promote the shared parameters aggregation process, we propose DFed integrating the local Sharpness Miniization.
  arXiv  Detail & Related papers  (2023-05-24T13:52:18Z)
• FedDWA: Personalized Federated Learning with Dynamic Weight Adjustment [20.72576355616359]
  We propose a new PFL algorithm called emphFedDWA (Federated Learning with Dynamic Weight Adjustment) to address the problem. FedDWA computes personalized aggregation weights based on collected models from clients. We conduct extensive experiments using five real datasets and the results demonstrate that FedDWA can significantly reduce the communication traffic and achieve much higher model accuracy than the state-of-the-art approaches.
  arXiv  Detail & Related papers  (2023-05-10T13:12:07Z)
• Personalized Federated Learning under Mixture of Distributions [98.25444470990107]
  We propose a novel approach to Personalized Federated Learning (PFL), which utilizes Gaussian mixture models (GMM) to fit the input data distributions across diverse clients. FedGMM possesses an additional advantage of adapting to new clients with minimal overhead, and it also enables uncertainty quantification. Empirical evaluations on synthetic and benchmark datasets demonstrate the superior performance of our method in both PFL classification and novel sample detection.
  arXiv  Detail & Related papers  (2023-05-01T20:04:46Z)
• PerAda: Parameter-Efficient Federated Learning Personalization with Generalization Guarantees [95.87604231887353]
  Existing pFL methods introduce high communication and computation costs or are vulnerable to test communication. In PerAda, a parameter distillation and pFL pFL has superior performance, especially under test-time distribution. Our code is available at https://github.com/NV/PerAda.
  arXiv  Detail & Related papers  (2023-02-13T19:00:37Z)
• Adaptive Parameterization of Deep Learning Models for Federated Learning [85.82002651944254]
  Federated Learning offers a way to train deep neural networks in a distributed fashion. It incurs a communication overhead as the model parameters or gradients need to be exchanged regularly during training. In this paper, we propose to utilise parallel Adapters for Federated Learning.
  arXiv  Detail & Related papers  (2023-02-06T17:30:33Z)
• Toward Understanding the Influence of Individual Clients in Federated Learning [52.07734799278535]
  Federated learning allows clients to jointly train a global model without sending their private data to a central server. We defined a new notion called em-Influence, quantify this influence over parameters, and proposed an effective efficient model to estimate this metric.
  arXiv  Detail & Related papers  (2020-12-20T14:34:36Z)
• Federated learning with hierarchical clustering of local updates to improve training on non-IID data [3.3517146652431378]
  We show that learning a single joint model is often not optimal in the presence of certain types of non-iid data. We present a modification to FL by introducing a hierarchical clustering step (FL+HC) We show how FL+HC allows model training to converge in fewer communication rounds compared to FL without clustering.
  arXiv  Detail & Related papers  (2020-04-24T15:16:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.