DualFed: Enjoying both Generalization and Personalization in Federated Learning via Hierachical Representations
- URL: http://arxiv.org/abs/2407.17754v1
- Date: Thu, 25 Jul 2024 04:09:12 GMT
- Title: DualFed: Enjoying both Generalization and Personalization in Federated Learning via Hierachical Representations
- Authors: Guogang Zhu, Xuefeng Liu, Jianwei Niu, Shaojie Tang, Xinghao Wu, Jiayuan Zhang,
- Abstract summary: In personalized federated learning (PFL), it is widely recognized that achieving both high model generalization and effective personalization poses a significant challenge.
Our paper presents an affirmative answer, and the key lies in the observation that deep models inherently exhibit hierarchical architectures.
We propose DualFed, a new method that can directly yield dual representations correspond to generalization and personalization respectively.
- Score: 12.941603966989364
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: In personalized federated learning (PFL), it is widely recognized that achieving both high model generalization and effective personalization poses a significant challenge due to their conflicting nature. As a result, existing PFL methods can only manage a trade-off between these two objectives. This raises an interesting question: Is it feasible to develop a model capable of achieving both objectives simultaneously? Our paper presents an affirmative answer, and the key lies in the observation that deep models inherently exhibit hierarchical architectures, which produce representations with various levels of generalization and personalization at different stages. A straightforward approach stemming from this observation is to select multiple representations from these layers and combine them to concurrently achieve generalization and personalization. However, the number of candidate representations is commonly huge, which makes this method infeasible due to high computational costs.To address this problem, we propose DualFed, a new method that can directly yield dual representations correspond to generalization and personalization respectively, thereby simplifying the optimization task. Specifically, DualFed inserts a personalized projection network between the encoder and classifier. The pre-projection representations are able to capture generalized information shareable across clients, and the post-projection representations are effective to capture task-specific information on local clients. This design minimizes the mutual interference between generalization and personalization, thereby achieving a win-win situation. Extensive experiments show that DualFed can outperform other FL methods. Code is available at https://github.com/GuogangZhu/DualFed.
Related papers
- Towards a Generalist and Blind RGB-X Tracker [91.36268768952755]
We develop a single model tracker that can remain blind to any modality X during inference time.
Our training process is extremely simple, integrating multi-label classification loss with a routing function.
Our generalist and blind tracker can achieve competitive performance compared to well-established modal-specific models.
arXiv Detail & Related papers (2024-05-28T03:00:58Z) - FedP3: Federated Personalized and Privacy-friendly Network Pruning under Model Heterogeneity [82.5448598805968]
We present an effective and adaptable federated framework FedP3, representing Federated Personalized and Privacy-friendly network Pruning.
We offer a theoretical interpretation of FedP3 and its locally differential-private variant, DP-FedP3, and theoretically validate their efficiencies.
arXiv Detail & Related papers (2024-04-15T14:14:05Z) - MAP: Model Aggregation and Personalization in Federated Learning with Incomplete Classes [49.22075916259368]
In some real-world applications, data samples are usually distributed on local devices.
In this paper, we focus on a special kind of Non-I.I.D. scene where clients own incomplete classes.
Our proposed algorithm named MAP could simultaneously achieve the aggregation and personalization goals in FL.
arXiv Detail & Related papers (2024-04-14T12:22:42Z) - FediOS: Decoupling Orthogonal Subspaces for Personalization in
Feature-skew Federated Learning [6.076894295435773]
In personalized federated learning (pFL), clients may have heterogeneous (also known as non-IID) data.
In FediOS, we reformulate the decoupling into two feature extractors (generic and personalized) and one shared prediction head.
In addition, a shared prediction head is trained to balance the importance of generic and personalized features during inference.
arXiv Detail & Related papers (2023-11-30T13:50:38Z) - FedBone: Towards Large-Scale Federated Multi-Task Learning [13.835972363413884]
In real-world applications, visual and natural language tasks typically require large-scale models to extract high-level abstract features.
Existing HFML methods disregard the impact of gradient conflicts on multi-task optimization.
We propose an innovative framework called FedBone, which enables the construction of large-scale models with better generalization.
arXiv Detail & Related papers (2023-06-30T08:19:38Z) - FedJETs: Efficient Just-In-Time Personalization with Federated Mixture
of Experts [48.78037006856208]
FedJETs is a novel solution by using a Mixture-of-Experts (MoE) framework within a Federated Learning (FL) setup.
Our method leverages the diversity of the clients to train specialized experts on different subsets of classes, and a gating function to route the input to the most relevant expert(s)
Our approach can improve accuracy up to 18% in state of the art FL settings, while maintaining competitive zero-shot performance.
arXiv Detail & Related papers (2023-06-14T15:47:52Z) - Personalization Disentanglement for Federated Learning: An explainable
perspective [28.780213981859514]
This paper addresses PFL via explicit disentangling latent representations into two parts to capture the shared knowledge and client-specific personalization.
The disentanglement is achieved by a novel Federated Dual Variational Autoencoder (FedDVA), which employs two encoders to infer the two types of representations.
arXiv Detail & Related papers (2023-06-06T10:37:11Z) - FedAvg with Fine Tuning: Local Updates Lead to Representation Learning [54.65133770989836]
Federated Averaging (FedAvg) algorithm consists of alternating between a few local gradient updates at client nodes, followed by a model averaging update at the server.
We show that the reason behind generalizability of the FedAvg's output is its power in learning the common data representation among the clients' tasks.
We also provide empirical evidence demonstrating FedAvg's representation learning ability in federated image classification with heterogeneous data.
arXiv Detail & Related papers (2022-05-27T00:55:24Z) - On Bridging Generic and Personalized Federated Learning [18.989191579101586]
We propose a novel federated learning framework that explicitly decouples a model's dual duties with two prediction tasks.
With this two-loss, two-predictor framework we name Federated Robust Decoupling Fed-RoD, the learned model can simultaneously achieve state-of-the-art generic and personalized performance.
arXiv Detail & Related papers (2021-07-02T00:25:48Z) - Personalized Federated Learning with Clustered Generalization [16.178571176116073]
We study the recent emerging personalized learning (PFL) that aims at dealing with the challenging problem of Non-I.I.D. data in the learning setting.
Key difference between PFL and conventional FL methods in the training target.
We propose a novel concept called clustered generalization to handle the challenge of statistical heterogeneity in FL.
arXiv Detail & Related papers (2021-06-24T14:17:00Z) - Federated Mutual Learning [65.46254760557073]
Federated Mutual Leaning (FML) allows clients training a generalized model collaboratively and a personalized model independently.
The experiments show that FML can achieve better performance than alternatives in typical Federated learning setting.
arXiv Detail & Related papers (2020-06-27T09:35:03Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.