FedIN: Federated Intermediate Layers Learning for Model Heterogeneity

• URL: http://arxiv.org/abs/2304.00759v3
• Date: Thu, 1 Feb 2024 10:40:09 GMT
• Title: FedIN: Federated Intermediate Layers Learning for Model Heterogeneity
• Authors: Yun-Hin Chan, Zhihan Jiang, Jing Deng, Edith C.-H. Ngai
• Abstract summary: Federated learning (FL) facilitates edge devices to cooperatively train a global shared model while maintaining the training data locally and privately. In this study, we propose an FL method called Federated Intermediate Layers Learning (FedIN), supporting heterogeneous models without relying on any public dataset. Experiment results demonstrate the superior performance of FedIN in heterogeneous model environments compared to state-of-the-art algorithms.
• Score: 7.781409257429762
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Federated learning (FL) facilitates edge devices to cooperatively train a global shared model while maintaining the training data locally and privately. However, a common assumption in FL requires the participating edge devices to have similar computation resources and train on an identical global model architecture. In this study, we propose an FL method called Federated Intermediate Layers Learning (FedIN), supporting heterogeneous models without relying on any public dataset. Instead, FedIN leverages the inherent knowledge embedded in client model features to facilitate knowledge exchange. The training models in FedIN are partitioned into three distinct components: an extractor, intermediate layers, and a classifier. We capture client features by extracting the outputs of the extractor and the inputs of the classifier. To harness the knowledge from client features, we propose IN training for aligning the intermediate layers based on features obtained from other clients. IN training only needs minimal memory and communication overhead by utilizing a single batch of client features. Additionally, we formulate and address a convex optimization problem to mitigate the challenge of gradient divergence caused by conflicts between IN training and local training. The experiment results demonstrate the superior performance of FedIN in heterogeneous model environments compared to state-of-the-art algorithms. Furthermore, our ablation study demonstrates the effectiveness of IN training and the proposed solution for alleviating gradient divergence.

Related papers

• Personalized Federated Learning via Feature Distribution Adaptation [3.410799378893257]
  Federated learning (FL) is a distributed learning framework that leverages commonalities between distributed client datasets to train a global model. personalized federated learning (PFL) seeks to address this by learning individual models tailored to each client. We propose an algorithm, pFedFDA, that efficiently generates personalized models by adapting global generative classifiers to their local feature distributions.
  arXiv  Detail & Related papers  (2024-11-01T03:03:52Z)
• Federated Learning under Partially Class-Disjoint Data via Manifold Reshaping [64.58402571292723]
  We propose a manifold reshaping approach called FedMR to calibrate the feature space of local training. We conduct extensive experiments on a range of datasets to demonstrate that our FedMR achieves much higher accuracy and better communication efficiency.
  arXiv  Detail & Related papers  (2024-05-29T10:56:13Z)
• pFedES: Model Heterogeneous Personalized Federated Learning with Feature Extractor Sharing [19.403843478569303]
  We propose a model-heterogeneous personalized Federated learning approach based on feature extractor sharing. It incorporates a small homogeneous feature extractor into each client's heterogeneous local model. It achieves 1.61% higher test accuracy, while reducing communication and computation costs by 99.6% and 82.9%, respectively.
  arXiv  Detail & Related papers  (2023-11-12T15:43:39Z)
• Effectively Heterogeneous Federated Learning: A Pairing and Split Learning Based Approach [16.093068118849246]
  This paper presents a novel split federated learning (SFL) framework that pairs clients with different computational resources. A greedy algorithm is proposed by reconstructing the optimization of training latency as a graph edge selection problem. Simulation results show the proposed method can significantly improve the FL training speed and achieve high performance.
  arXiv  Detail & Related papers  (2023-08-26T11:10:54Z)
• Towards Instance-adaptive Inference for Federated Learning [80.38701896056828]
  Federated learning (FL) is a distributed learning paradigm that enables multiple clients to learn a powerful global model by aggregating local training. In this paper, we present a novel FL algorithm, i.e., FedIns, to handle intra-client data heterogeneity by enabling instance-adaptive inference in the FL framework. Our experiments show that our FedIns outperforms state-of-the-art FL algorithms, e.g., a 6.64% improvement against the top-performing method with less than 15% communication cost on Tiny-ImageNet.
  arXiv  Detail & Related papers  (2023-08-11T09:58:47Z)
• Personalizing Federated Learning with Over-the-Air Computations [84.8089761800994]
  Federated edge learning is a promising technology to deploy intelligence at the edge of wireless networks in a privacy-preserving manner. Under such a setting, multiple clients collaboratively train a global generic model under the coordination of an edge server. This paper presents a distributed training paradigm that employs analog over-the-air computation to address the communication bottleneck.
  arXiv  Detail & Related papers  (2023-02-24T08:41:19Z)
• Federated Learning with Intermediate Representation Regularization [14.01585596739954]
  Federated learning (FL) enables remote clients to collaboratively train a model without exposing their private data. Previous studies accomplish this by regularizing the distance between the representations learned by the local and global models. We introduce FedIntR, which provides a more fine-grained regularization by integrating the representations of intermediate layers into the local training process.
  arXiv  Detail & Related papers  (2022-10-28T01:43:55Z)
• FedClassAvg: Local Representation Learning for Personalized Federated Learning on Heterogeneous Neural Networks [21.613436984547917]
  We propose a novel personalized federated learning method called federated classifier averaging (FedClassAvg) FedClassAvg aggregates weights as an agreement on decision boundaries on feature spaces. We demonstrate it outperforms the current state-of-the-art algorithms on heterogeneous personalized federated learning tasks.
  arXiv  Detail & Related papers  (2022-10-25T08:32:08Z)
• FedDM: Iterative Distribution Matching for Communication-Efficient Federated Learning [87.08902493524556]
  Federated learning(FL) has recently attracted increasing attention from academia and industry. We propose FedDM to build the global training objective from multiple local surrogate functions. In detail, we construct synthetic sets of data on each client to locally match the loss landscape from original data.
  arXiv  Detail & Related papers  (2022-07-20T04:55:18Z)
• Toward Understanding the Influence of Individual Clients in Federated Learning [52.07734799278535]
  Federated learning allows clients to jointly train a global model without sending their private data to a central server. We defined a new notion called em-Influence, quantify this influence over parameters, and proposed an effective efficient model to estimate this metric.
  arXiv  Detail & Related papers  (2020-12-20T14:34:36Z)
• Federated Residual Learning [53.77128418049985]
  We study a new form of federated learning where the clients train personalized local models and make predictions jointly with the server-side shared model. Using this new federated learning framework, the complexity of the central shared model can be minimized while still gaining all the performance benefits that joint training provides.
  arXiv  Detail & Related papers  (2020-03-28T19:55:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.