FedHCA$^2$: Towards Hetero-Client Federated Multi-Task Learning
- URL: http://arxiv.org/abs/2311.13250v2
- Date: Thu, 29 Feb 2024 03:32:35 GMT
- Title: FedHCA$^2$: Towards Hetero-Client Federated Multi-Task Learning
- Authors: Yuxiang Lu, Suizhi Huang, Yuwen Yang, Shalayiding Sirejiding, Yue
Ding, Hongtao Lu
- Abstract summary: Federated Learning (FL) enables joint training across distributed clients using their local data privately.
We introduce a novel problem setting, Hetero-Client Federated Multi-Task Learning (HC-FMTL), to accommodate diverse task setups.
We propose the FedHCA$2$ framework, which allows for federated training of personalized models by modeling relationships among heterogeneous clients.
- Score: 18.601886059536326
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Federated Learning (FL) enables joint training across distributed clients
using their local data privately. Federated Multi-Task Learning (FMTL) builds
on FL to handle multiple tasks, assuming model congruity that identical model
architecture is deployed in each client. To relax this assumption and thus
extend real-world applicability, we introduce a novel problem setting,
Hetero-Client Federated Multi-Task Learning (HC-FMTL), to accommodate diverse
task setups. The main challenge of HC-FMTL is the model incongruity issue that
invalidates conventional aggregation methods. It also escalates the
difficulties in accurate model aggregation to deal with data and task
heterogeneity inherent in FMTL. To address these challenges, we propose the
FedHCA$^2$ framework, which allows for federated training of personalized
models by modeling relationships among heterogeneous clients. Drawing on our
theoretical insights into the difference between multi-task and federated
optimization, we propose the Hyper Conflict-Averse Aggregation scheme to
mitigate conflicts during encoder updates. Additionally, inspired by task
interaction in MTL, the Hyper Cross Attention Aggregation scheme uses
layer-wise cross attention to enhance decoder interactions while alleviating
model incongruity. Moreover, we employ learnable Hyper Aggregation Weights for
each client to customize personalized parameter updates. Extensive experiments
demonstrate the superior performance of FedHCA$^2$ in various HC-FMTL scenarios
compared to representative methods. Our code will be made publicly available.
Related papers
- Adaptive Prototype Knowledge Transfer for Federated Learning with Mixed Modalities and Heterogeneous Tasks [12.67996108615162]
We propose an Adaptive prototype-based Multimodal Federated Learning (AproMFL) framework for mixed modalities and heterogeneous tasks.
Our AproMFL transfers knowledge through adaptively-constructed prototypes without a prior public dataset.
Clients adaptively select prototype construction methods in line with tasks; server converts client prototypes into unified multimodal prototypes and aggregates them to form global prototypes.
arXiv Detail & Related papers (2025-02-06T07:28:05Z) - Tackling Feature and Sample Heterogeneity in Decentralized Multi-Task Learning: A Sheaf-Theoretic Approach [37.4602828056364]
We introduce a novel sheaf-theoretic-based approach for Federated Multi-task Learning (FMTL)
By representing client relationships using cellular sheaves, our framework can flexibly model interactions between heterogeneous client models.
We show that our proposed algorithm, Sheaf-FMTL, achieves a sublinear convergence rate in line with state-of-the-art decentralized FMTL algorithms.
arXiv Detail & Related papers (2025-02-03T08:25:34Z) - Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descent [74.02034188307857]
Merging multiple expert models offers a promising approach for performing multi-task learning without accessing their original data.
We find existing methods inevitably discard task-specific information that, while causing conflicts, is crucial for performance.
Our approach consistently outperforms previous methods, achieving state-of-the-art results across diverse architectures and tasks in both vision and NLP domains.
arXiv Detail & Related papers (2025-01-02T12:45:21Z) - FedMoE: Personalized Federated Learning via Heterogeneous Mixture of Experts [4.412721048192925]
We present FedMoE, the efficient personalized Federated Learning framework to address data heterogeneity.
FedMoE is composed of two fine-tuning stages. In the first stage, FedMoE simplifies the problem by conducting a search based on observed activation patterns.
In the second stage, these submodels are distributed to clients for further training and returned for server aggregating.
arXiv Detail & Related papers (2024-08-21T03:16:12Z) - Leveraging Foundation Models for Multi-modal Federated Learning with Incomplete Modality [41.79433449873368]
We propose a novel multi-modal federated learning method, Federated Multi-modal contrastiVe training with Pre-trained completion (FedMVP)
FedMVP integrates the large-scale pre-trained models to enhance the federated training.
We demonstrate that the model achieves superior performance over two real-world image-text classification datasets.
arXiv Detail & Related papers (2024-06-16T19:18:06Z) - Merging Multi-Task Models via Weight-Ensembling Mixture of Experts [64.94129594112557]
Merging Transformer-based models trained on different tasks into a single unified model can execute all the tasks concurrently.
Previous methods, exemplified by task arithmetic, have been proven to be both effective and scalable.
We propose to merge most of the parameters while upscaling the Transformer layers to a weight-ensembling mixture of experts (MoE) module.
arXiv Detail & Related papers (2024-02-01T08:58:57Z) - FedYolo: Augmenting Federated Learning with Pretrained Transformers [61.56476056444933]
In this work, we investigate pretrained transformers (PTF) to achieve on-device learning goals.
We show that larger scale shrinks the accuracy gaps between alternative approaches and improves robustness.
Finally, it enables clients to solve multiple unrelated tasks simultaneously using a single PTF.
arXiv Detail & Related papers (2023-07-10T21:08:52Z) - FedBone: Towards Large-Scale Federated Multi-Task Learning [13.835972363413884]
In real-world applications, visual and natural language tasks typically require large-scale models to extract high-level abstract features.
Existing HFML methods disregard the impact of gradient conflicts on multi-task optimization.
We propose an innovative framework called FedBone, which enables the construction of large-scale models with better generalization.
arXiv Detail & Related papers (2023-06-30T08:19:38Z) - Mod-Squad: Designing Mixture of Experts As Modular Multi-Task Learners [74.92558307689265]
We propose Mod-Squad, a new model that is Modularized into groups of experts (a 'Squad')
We optimize this matching process during the training of a single model.
Experiments on the Taskonomy dataset with 13 vision tasks and the PASCAL-Context dataset with 5 vision tasks show the superiority of our approach.
arXiv Detail & Related papers (2022-12-15T18:59:52Z) - FedGradNorm: Personalized Federated Gradient-Normalized Multi-Task
Learning [50.756991828015316]
Multi-task learning (MTL) is a novel framework to learn several tasks simultaneously with a single shared network.
We propose FedGradNorm which uses a dynamic-weighting method to normalize norms in order to balance learning speeds among different tasks.
arXiv Detail & Related papers (2022-03-24T17:43:12Z) - Federated Mutual Learning [65.46254760557073]
Federated Mutual Leaning (FML) allows clients training a generalized model collaboratively and a personalized model independently.
The experiments show that FML can achieve better performance than alternatives in typical Federated learning setting.
arXiv Detail & Related papers (2020-06-27T09:35:03Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.