FedBone: Towards Large-Scale Federated Multi-Task Learning

• URL: http://arxiv.org/abs/2306.17465v1
• Date: Fri, 30 Jun 2023 08:19:38 GMT
• Title: FedBone: Towards Large-Scale Federated Multi-Task Learning
• Authors: Yiqiang Chen, Teng Zhang, Xinlong Jiang, Qian Chen, Chenlong Gao and Wuliang Huang
• Abstract summary: In real-world applications, visual and natural language tasks typically require large-scale models to extract high-level abstract features. Existing HFML methods disregard the impact of gradient conflicts on multi-task optimization. We propose an innovative framework called FedBone, which enables the construction of large-scale models with better generalization.
• Score: 13.835972363413884
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Heterogeneous federated multi-task learning (HFMTL) is a federated learning technique that combines heterogeneous tasks of different clients to achieve more accurate, comprehensive predictions. In real-world applications, visual and natural language tasks typically require large-scale models to extract high-level abstract features. However, large-scale models cannot be directly applied to existing federated multi-task learning methods. Existing HFML methods also disregard the impact of gradient conflicts on multi-task optimization during the federated aggregation process. In this work, we propose an innovative framework called FedBone, which enables the construction of large-scale models with better generalization from the perspective of server-client split learning and gradient projection. We split the entire model into two components: a large-scale general model (referred to as the general model) on the cloud server and multiple task-specific models (referred to as the client model) on edge clients, solving the problem of insufficient computing power on edge clients. The conflicting gradient projection technique is used to enhance the generalization of the large-scale general model between different tasks. The proposed framework is evaluated on two benchmark datasets and a real ophthalmic dataset. Comprehensive results demonstrate that FedBone efficiently adapts to heterogeneous local tasks of each client and outperforms existing federated learning algorithms in most dense prediction and classification tasks with off-the-shelf computational resources on the client side.

Related papers

• Leveraging Foundation Models for Multi-modal Federated Learning with Incomplete Modality [41.79433449873368]
  We propose a novel multi-modal federated learning method, Federated Multi-modal contrastiVe training with Pre-trained completion (FedMVP) FedMVP integrates the large-scale pre-trained models to enhance the federated training. We demonstrate that the model achieves superior performance over two real-world image-text classification datasets.
  arXiv  Detail & Related papers  (2024-06-16T19:18:06Z)
• Towards Efficient Pareto Set Approximation via Mixture of Experts Based Model Fusion [53.33473557562837]
  Solving multi-objective optimization problems for large deep neural networks is a challenging task due to the complexity of the loss landscape and the expensive computational cost. We propose a practical and scalable approach to solve this problem via mixture of experts (MoE) based model fusion. By ensembling the weights of specialized single-task models, the MoE module can effectively capture the trade-offs between multiple objectives.
  arXiv  Detail & Related papers  (2024-06-14T07:16:18Z)
• Concrete Subspace Learning based Interference Elimination for Multi-task Model Fusion [86.6191592951269]
  Merging models fine-tuned from common extensively pretrained large model but specialized for different tasks has been demonstrated as a cheap and scalable strategy to construct a multitask model that performs well across diverse tasks. We propose the CONtinuous relaxation dis (Concrete) subspace learning method to identify a common lowdimensional subspace and utilize its shared information track interference problem without sacrificing performance.
  arXiv  Detail & Related papers  (2023-12-11T07:24:54Z)
• FedHCA$^2$: Towards Hetero-Client Federated Multi-Task Learning [18.601886059536326]
  Federated Learning (FL) enables joint training across distributed clients using their local data privately. We introduce a novel problem setting, Hetero-Client Federated Multi-Task Learning (HC-FMTL), to accommodate diverse task setups. We propose the FedHCA$2$ framework, which allows for federated training of personalized models by modeling relationships among heterogeneous clients.
  arXiv  Detail & Related papers  (2023-11-22T09:12:50Z)
• RGM: A Robust Generalizable Matching Model [49.60975442871967]
  We propose a deep model for sparse and dense matching, termed RGM (Robust Generalist Matching) To narrow the gap between synthetic training samples and real-world scenarios, we build a new, large-scale dataset with sparse correspondence ground truth. We are able to mix up various dense and sparse matching datasets, significantly improving the training diversity.
  arXiv  Detail & Related papers  (2023-10-18T07:30:08Z)
• Personalized Federated Learning with Feature Alignment and Classifier Collaboration [13.320381377599245]
  Data heterogeneity is one of the most challenging issues in federated learning. One such approach in deep neural networks based tasks is employing a shared feature representation and learning a customized classifier head for each client. In this work, we conduct explicit local-global feature alignment by leveraging global semantic knowledge for learning a better representation.
  arXiv  Detail & Related papers  (2023-06-20T19:58:58Z)
• Multimodal Federated Learning via Contrastive Representation Ensemble [17.08211358391482]
  Federated learning (FL) serves as a privacy-conscious alternative to centralized machine learning. Existing FL methods all rely on model aggregation on single modality level. We propose Contrastive Representation Ensemble and Aggregation for Multimodal FL (CreamFL)
  arXiv  Detail & Related papers  (2023-02-17T14:17:44Z)
• Heterogeneous Ensemble Knowledge Transfer for Training Large Models in Federated Learning [22.310090483499035]
  Federated learning (FL) enables edge-devices to collaboratively learn a model without disclosing their private data to a central aggregating server. Most existing FL algorithms require models of identical architecture to be deployed across the clients and server. We propose a novel ensemble knowledge transfer method named Fed-ET in which small models are trained on clients, and used to train a larger model at the server.
  arXiv  Detail & Related papers  (2022-04-27T05:18:32Z)
• No One Left Behind: Inclusive Federated Learning over Heterogeneous Devices [79.16481453598266]
  We propose InclusiveFL, a client-inclusive federated learning method to handle this problem. The core idea of InclusiveFL is to assign models of different sizes to clients with different computing capabilities. We also propose an effective method to share the knowledge among multiple local models with different sizes.
  arXiv  Detail & Related papers  (2022-02-16T13:03:27Z)
• An Expectation-Maximization Perspective on Federated Learning [75.67515842938299]
  Federated learning describes the distributed training of models across multiple clients while keeping the data private on-device. In this work, we view the server-orchestrated federated learning process as a hierarchical latent variable model where the server provides the parameters of a prior distribution over the client-specific model parameters. We show that with simple Gaussian priors and a hard version of the well known Expectation-Maximization (EM) algorithm, learning in such a model corresponds to FedAvg, the most popular algorithm for the federated learning setting.
  arXiv  Detail & Related papers  (2021-11-19T12:58:59Z)
• Exploiting Shared Representations for Personalized Federated Learning [54.65133770989836]
  We propose a novel federated learning framework and algorithm for learning a shared data representation across clients and unique local heads for each client. Our algorithm harnesses the distributed computational power across clients to perform many local-updates with respect to the low-dimensional local parameters for every update of the representation. This result is of interest beyond federated learning to a broad class of problems in which we aim to learn a shared low-dimensional representation among data distributions.
  arXiv  Detail & Related papers  (2021-02-14T05:36:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.