FedCLIP: Fast Generalization and Personalization for CLIP in Federated Learning

• URL: http://arxiv.org/abs/2302.13485v2
• Date: Sun, 9 Jul 2023 12:36:50 GMT
• Title: FedCLIP: Fast Generalization and Personalization for CLIP in Federated Learning
• Authors: Wang Lu, Xixu Hu, Jindong Wang, Xing Xie
• Abstract summary: Federated learning (FL) has emerged as a new paradigm for privacy-preserving computation in recent years. FL faces two critical challenges that hinder its actual performance: data distribution Heterogeneous and high resource costs. We propose FedCLIP to achieve fast generalization and personalization for CLIP in FL.
• Score: 18.763298147996238
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Federated learning (FL) has emerged as a new paradigm for privacy-preserving computation in recent years. Unfortunately, FL faces two critical challenges that hinder its actual performance: data distribution heterogeneity and high resource costs brought by large foundation models. Specifically, the non-IID data in different clients make existing FL algorithms hard to converge while the high resource costs, including computational and communication costs that increase the deployment difficulty in real-world scenarios. In this paper, we propose an effective yet simple method, named FedCLIP, to achieve fast generalization and personalization for CLIP in federated learning. Concretely, we design an attention-based adapter for the large model, CLIP, and the rest operations merely depend on adapters. Lightweight adapters can make the most use of pretrained model information and ensure models be adaptive for clients in specific tasks. Simultaneously, small-scale operations can mitigate the computational burden and communication burden caused by large models. Extensive experiments are conducted on three datasets with distribution shifts. Qualitative and quantitative results demonstrate that FedCLIP significantly outperforms other baselines (9% overall improvements on PACS) and effectively reduces computational and communication costs (283x faster than FedAVG). Our code will be available at: https://github.com/microsoft/PersonalizedFL.

Related papers

• TriplePlay: Enhancing Federated Learning with CLIP for Non-IID Data and Resource Efficiency [0.0]
  TriplePlay is a framework that integrates CLIP as an adapter to enhance FL's adaptability and performance across diverse data distributions. Our simulation results demonstrate that TriplePlay effectively decreases GPU usage costs and speeds up the learning process, achieving convergence with reduced communication overhead.
  arXiv  Detail & Related papers  (2024-09-09T06:04:42Z)
• Lightweight Industrial Cohorted Federated Learning for Heterogeneous Assets [0.0]
  Federated Learning (FL) is the most widely adopted collaborative learning approach for training decentralized Machine Learning (ML) models. However, since great data similarity or homogeneity is taken for granted in all FL tasks, FL is still not specifically designed for the industrial setting. We propose a Lightweight Industrial Cohorted FL (LICFL) algorithm that uses model parameters for cohorting without any additional on-edge (clientlevel) computations and communications.
  arXiv  Detail & Related papers  (2024-07-25T12:48:56Z)
• Training Heterogeneous Client Models using Knowledge Distillation in Serverless Federated Learning [0.5510212613486574]
  Federated Learning (FL) is an emerging machine learning paradigm that enables the collaborative training of a shared global model across distributed clients. Recent works on designing systems for efficient FL have shown that utilizing serverless computing technologies can enhance resource efficiency, reduce training costs, and alleviate the complex infrastructure management burden on data holders.
  arXiv  Detail & Related papers  (2024-02-11T20:15:52Z)
• Fed-CVLC: Compressing Federated Learning Communications with Variable-Length Codes [54.18186259484828]
  In Federated Learning (FL) paradigm, a parameter server (PS) concurrently communicates with distributed participating clients for model collection, update aggregation, and model distribution over multiple rounds. We show strong evidences that variable-length is beneficial for compression in FL. We present Fed-CVLC (Federated Learning Compression with Variable-Length Codes), which fine-tunes the code length in response to the dynamics of model updates.
  arXiv  Detail & Related papers  (2024-02-06T07:25:21Z)
• ZooPFL: Exploring Black-box Foundation Models for Personalized Federated Learning [95.64041188351393]
  This paper endeavors to solve both the challenges of limited resources and personalization. We propose a method named ZOOPFL that uses Zeroth-Order Optimization for Personalized Federated Learning. To reduce the computation costs and enhance personalization, we propose input surgery to incorporate an auto-encoder with low-dimensional and client-specific embeddings.
  arXiv  Detail & Related papers  (2023-10-08T12:26:13Z)
• FedLALR: Client-Specific Adaptive Learning Rates Achieve Linear Speedup for Non-IID Data [54.81695390763957]
  Federated learning is an emerging distributed machine learning method. We propose a heterogeneous local variant of AMSGrad, named FedLALR, in which each client adjusts its learning rate. We show that our client-specified auto-tuned learning rate scheduling can converge and achieve linear speedup with respect to the number of clients.
  arXiv  Detail & Related papers  (2023-09-18T12:35:05Z)
• Towards Instance-adaptive Inference for Federated Learning [80.38701896056828]
  Federated learning (FL) is a distributed learning paradigm that enables multiple clients to learn a powerful global model by aggregating local training. In this paper, we present a novel FL algorithm, i.e., FedIns, to handle intra-client data heterogeneity by enabling instance-adaptive inference in the FL framework. Our experiments show that our FedIns outperforms state-of-the-art FL algorithms, e.g., a 6.64% improvement against the top-performing method with less than 15% communication cost on Tiny-ImageNet.
  arXiv  Detail & Related papers  (2023-08-11T09:58:47Z)
• User-Centric Federated Learning: Trading off Wireless Resources for Personalization [18.38078866145659]
  In Federated Learning (FL) systems, Statistical Heterogeneousness increases the algorithm convergence time and reduces the generalization performance. To tackle the above problems without violating the privacy constraints that FL imposes, personalized FL methods have to couple statistically similar clients without directly accessing their data. In this work, we design user-centric aggregation rules that are based on readily available gradient information and are capable of producing personalized models for each FL client. Our algorithm outperforms popular personalized FL baselines in terms of average accuracy, worst node performance, and training communication overhead.
  arXiv  Detail & Related papers  (2023-04-25T15:45:37Z)
• PerAda: Parameter-Efficient Federated Learning Personalization with Generalization Guarantees [95.87604231887353]
  Existing pFL methods introduce high communication and computation costs or are vulnerable to test communication. In PerAda, a parameter distillation and pFL pFL has superior performance, especially under test-time distribution. Our code is available at https://github.com/NV/PerAda.
  arXiv  Detail & Related papers  (2023-02-13T19:00:37Z)
• Dynamic Attention-based Communication-Efficient Federated Learning [85.18941440826309]
  Federated learning (FL) offers a solution to train a global machine learning model. FL suffers performance degradation when client data distribution is non-IID. We propose a new adaptive training algorithm $textttAdaFL$ to combat this degradation.
  arXiv  Detail & Related papers  (2021-08-12T14:18:05Z)
• Personalized Federated Learning via Maximizing Correlation with Sparse and Hierarchical Extensions [14.862798952297105]
  Federated Learning (FL) is a collaborative machine learning technique to train a global model without obtaining clients' private data. We propose a novel personalized federated learning via maximizing correlation pFedMac. We show that pFedMac performs better than the L2-norm distance based personalization methods.
  arXiv  Detail & Related papers  (2021-07-12T11:43:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.