Effective Heterogeneous Federated Learning via Efficient Hypernetwork-based Weight Generation

• URL: http://arxiv.org/abs/2407.03086v2
• Date: Thu, 03 Oct 2024 12:45:48 GMT
• Title: Effective Heterogeneous Federated Learning via Efficient Hypernetwork-based Weight Generation
• Authors: Yujin Shin, Kichang Lee, Sungmin Lee, You Rim Choi, Hyung-Sin Kim, JeongGil Ko,
• Abstract summary: We propose HypeMeFed, a novel federated learning framework for supporting client heterogeneity. This approach aligns the feature spaces of heterogeneous model layers and resolves per-layer information disparity during weight aggregation. Our evaluations on a real-world heterogeneous device testbed indicate that system enhances accuracy by 5.12% over FedAvg, reduces the hypernetwork memory requirements by 98.22%, and accelerates its operations by 1.86x compared to a naive hypernetwork approach.
• Score: 5.3125752371246575
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While federated learning leverages distributed client resources, it faces challenges due to heterogeneous client capabilities. This necessitates allocating models suited to clients' resources and careful parameter aggregation to accommodate this heterogeneity. We propose HypeMeFed, a novel federated learning framework for supporting client heterogeneity by combining a multi-exit network architecture with hypernetwork-based model weight generation. This approach aligns the feature spaces of heterogeneous model layers and resolves per-layer information disparity during weight aggregation. To practically realize HypeMeFed, we also propose a low-rank factorization approach to minimize computation and memory overhead associated with hypernetworks. Our evaluations on a real-world heterogeneous device testbed indicate that \system enhances accuracy by 5.12% over FedAvg, reduces the hypernetwork memory requirements by 98.22%, and accelerates its operations by 1.86x compared to a naive hypernetwork approach. These results demonstrate HypeMeFed's effectiveness in leveraging and engaging heterogeneous clients for federated learning.

Related papers

• Tackling Resource-Constrained and Data-Heterogeneity in Federated Learning with Double-Weight Sparse Pack [23.772570707484746]
  Federated learning has drawn widespread interest from researchers, yet the data heterogeneity across edge clients remains a key challenge.<n>We propose a personalized federated learning method based on cosine sparsification parameter packing and dual-weighted aggregation.
  arXiv  Detail & Related papers  (2026-01-05T07:03:04Z)
• Deploying Models to Non-participating Clients in Federated Learning without Fine-tuning: A Hypernetwork-based Approach [22.030687488408496]
  Federated Learning (FL) has emerged as a promising paradigm for privacy-preserving collaborative learning.<n>We present HyperFedZero, a novel method that dynamically generates specialized models via a hypernetwork conditioned on distribution-aware embeddings.
  arXiv  Detail & Related papers  (2025-08-18T07:11:51Z)
• Hypernetworks for Model-Heterogeneous Personalized Federated Learning [13.408669475480824]
  We propose a server-side hypernetwork that takes client-specific embedding vectors as input and outputs personalized parameters tailored to each client's heterogeneous model.<n>To promote knowledge sharing and reduce computation, we introduce a multi-head structure within the hypernetwork, allowing clients with similar model sizes to share heads.<n>Our framework does not rely on external datasets and does not require disclosure of client model architectures.
  arXiv  Detail & Related papers  (2025-07-30T02:24:26Z)
• Efficient Federated Learning with Heterogeneous Data and Adaptive Dropout [62.73150122809138]
  Federated Learning (FL) is a promising distributed machine learning approach that enables collaborative training of a global model using multiple edge devices.<n>We propose the FedDHAD FL framework, which comes with two novel methods: Dynamic Heterogeneous model aggregation (FedDH) and Adaptive Dropout (FedAD)<n>The combination of these two methods makes FedDHAD significantly outperform state-of-the-art solutions in terms of accuracy (up to 6.7% higher), efficiency (up to 2.02 times faster), and cost (up to 15.0% smaller)
  arXiv  Detail & Related papers  (2025-07-14T16:19:00Z)
• Client-Centric Federated Adaptive Optimization [78.30827455292827]
  Federated Learning (FL) is a distributed learning paradigm where clients collaboratively train a model while keeping their own data private. We propose Federated-Centric Adaptive Optimization, which is a class of novel federated optimization approaches.
  arXiv  Detail & Related papers  (2025-01-17T04:00:50Z)
• Over-the-Air Fair Federated Learning via Multi-Objective Optimization [52.295563400314094]
  We propose an over-the-air fair federated learning algorithm (OTA-FFL) to train fair FL models. Experiments demonstrate the superiority of OTA-FFL in achieving fairness and robust performance.
  arXiv  Detail & Related papers  (2025-01-06T21:16:51Z)
• Robust Federated Learning in the Face of Covariate Shift: A Magnitude Pruning with Hybrid Regularization Framework for Enhanced Model Aggregation [1.519321208145928]
  Federated Learning (FL) offers a promising framework for individuals aiming to collaboratively develop a shared model. variations in data distribution among clients can profoundly affect FL methodologies, primarily due to instabilities in the aggregation process. We propose a novel FL framework, combining individual parameter pruning and regularization techniques to improve the robustness of individual clients' models to aggregate.
  arXiv  Detail & Related papers  (2024-12-19T16:22:37Z)
• HeteroTune: Efficient Federated Learning for Large Heterogeneous Models [35.53420882449293]
  We propose HeteroTune, a novel federated fine-tuning paradigm for large, heterogeneous models operating under limited communication and budgets.<n>The core of our method lies in a novel architecture, DeMA, which enables flexible and efficient aggregation of heterogeneous models.<n>We provide both theoretical analysis and empirical evidence showing that HeteroTune achieves state-of-the-art performance and efficiency across diverse tasks and model architectures.
  arXiv  Detail & Related papers  (2024-11-25T09:58:51Z)
• Non-Federated Multi-Task Split Learning for Heterogeneous Sources [17.47679789733922]
  We introduce a new architecture and methodology to perform multi-task learning for heterogeneous data sources efficiently. We show through theoretical analysis that MTSL can achieve fast convergence by tuning the learning rate of the server and clients.
  arXiv  Detail & Related papers  (2024-05-31T19:27:03Z)
• Efficient Model Compression for Hierarchical Federated Learning [10.37403547348343]
  Federated learning (FL) has garnered significant attention due to its capacity to preserve privacy within distributed learning systems. This paper introduces a novel hierarchical FL framework that integrates the benefits of clustered FL and model compression.
  arXiv  Detail & Related papers  (2024-05-27T12:17:47Z)
• An Aggregation-Free Federated Learning for Tackling Data Heterogeneity [50.44021981013037]
  Federated Learning (FL) relies on the effectiveness of utilizing knowledge from distributed datasets. Traditional FL methods adopt an aggregate-then-adapt framework, where clients update local models based on a global model aggregated by the server from the previous training round. We introduce FedAF, a novel aggregation-free FL algorithm.
  arXiv  Detail & Related papers  (2024-04-29T05:55:23Z)
• FLASH: Federated Learning Across Simultaneous Heterogeneities [54.80435317208111]
  FLASH(Federated Learning Across Simultaneous Heterogeneities) is a lightweight and flexible client selection algorithm. It outperforms state-of-the-art FL frameworks under extensive sources of Heterogeneities. It achieves substantial and consistent improvements over state-of-the-art baselines.
  arXiv  Detail & Related papers  (2024-02-13T20:04:39Z)
• Federated cINN Clustering for Accurate Clustered Federated Learning [33.72494731516968]
  Federated Learning (FL) presents an innovative approach to privacy-preserving distributed machine learning. We propose the Federated cINN Clustering Algorithm (FCCA) to robustly cluster clients into different groups.
  arXiv  Detail & Related papers  (2023-09-04T10:47:52Z)
• Towards Instance-adaptive Inference for Federated Learning [80.38701896056828]
  Federated learning (FL) is a distributed learning paradigm that enables multiple clients to learn a powerful global model by aggregating local training. In this paper, we present a novel FL algorithm, i.e., FedIns, to handle intra-client data heterogeneity by enabling instance-adaptive inference in the FL framework. Our experiments show that our FedIns outperforms state-of-the-art FL algorithms, e.g., a 6.64% improvement against the top-performing method with less than 15% communication cost on Tiny-ImageNet.
  arXiv  Detail & Related papers  (2023-08-11T09:58:47Z)
• Tackling Computational Heterogeneity in FL: A Few Theoretical Insights [68.8204255655161]
  We introduce and analyse a novel aggregation framework that allows for formalizing and tackling computational heterogeneous data. Proposed aggregation algorithms are extensively analyzed from a theoretical, and an experimental prospective.
  arXiv  Detail & Related papers  (2023-07-12T16:28:21Z)
• Personalizing Federated Learning with Over-the-Air Computations [84.8089761800994]
  Federated edge learning is a promising technology to deploy intelligence at the edge of wireless networks in a privacy-preserving manner. Under such a setting, multiple clients collaboratively train a global generic model under the coordination of an edge server. This paper presents a distributed training paradigm that employs analog over-the-air computation to address the communication bottleneck.
  arXiv  Detail & Related papers  (2023-02-24T08:41:19Z)
• On the effectiveness of partial variance reduction in federated learning with heterogeneous data [27.527995694042506]
  We show that the diversity of the final classification layers across clients impedes the performance of the FedAvg algorithm. Motivated by this, we propose to correct model by variance reduction only on the final layers. We demonstrate that this significantly outperforms existing benchmarks at a similar or lower communication cost.
  arXiv  Detail & Related papers  (2022-12-05T11:56:35Z)
• Genetic CFL: Optimization of Hyper-Parameters in Clustered Federated Learning [4.710427287359642]
  Federated learning (FL) is a distributed model for deep learning that integrates client-server architecture, edge computing, and real-time intelligence. FL has the capability of revolutionizing machine learning (ML) but lacks in the practicality of implementation due to technological limitations, communication overhead, non-IID (independent and identically distributed) data, and privacy concerns. We propose a novel hybrid algorithm, namely genetic clustered FL (Genetic CFL), that clusters edge devices based on the training hyper- parameters and genetically modifies the parameters cluster-wise.
  arXiv  Detail & Related papers  (2021-07-15T10:16:05Z)
• Towards Fair Federated Learning with Zero-Shot Data Augmentation [123.37082242750866]
  Federated learning has emerged as an important distributed learning paradigm, where a server aggregates a global model from many client-trained models while having no access to the client data. We propose a novel federated learning system that employs zero-shot data augmentation on under-represented data to mitigate statistical heterogeneity and encourage more uniform accuracy performance across clients in federated networks. We study two variants of this scheme, Fed-ZDAC (federated learning with zero-shot data augmentation at the clients) and Fed-ZDAS (federated learning with zero-shot data augmentation at the server).
  arXiv  Detail & Related papers  (2021-04-27T18:23:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.