FEAST: Federated Shared-Space Training for Resource-Heterogeneous Clients

• URL: http://arxiv.org/abs/2608.09250v1
• Date: Mon, 10 Aug 2026 08:10:59 GMT
• Title: FEAST: Federated Shared-Space Training for Resource-Heterogeneous Clients
• Abstract summary: Federated learning must serve devices with varying computational capabilities.<n>Supernet training instead learns one fixed elastic model with differently sizedteroworks, then deploys a suitable one to each device.<n>When client inference budgets differ, parameters exclusive to high-costworks are reachable by fewer clients.<n>We propose FEAST, a shared-space training framework that counters this imbalance by jointly training multipleworks within each client's limit.
• Score: 2.1485350418225244
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Federated learning (FL) must serve devices with varying computational capabilities. A fixed model cannot suit all devices, while training one model per deployment limit is costly. Federated supernet training instead learns one elastic model with differently sized subnetworks, then deploys a suitable one to each device. When client inference budgets differ, however, parameters exclusive to high-cost subnetworks are reachable by fewer clients. We propose FEAST, a federated shared-space training framework that counters this imbalance by jointly training multiple subnetworks within each client's limit. Budget-tailored sub-supernet routing sends only the relevant supernet portion, and sparse aggregation merges the returned parameter slices. The trained supernet directly serves the subnetworks used during federation and supports post-hoc extraction of additional subnetworks without federated retraining. We further show that independently assigning clients' training-data volumes and inference budgets can distort accuracy--inference-cost comparisons in heterogeneous FL simulations, and introduce a one-parameter $γ$-allocation protocol to control this coupling. In our experimental setup, the SuperFedNAS and DeepFedNAS supernet training procedures remain near chance at 25M and reach at most $17.09\%$ at $596$M inference MACs; FEAST reaches $71.06\%$ at $596$M, $2.4$ points above the strongest model-heterogeneous weight-sharing baseline at its largest tier. Across CIFAR-100, CINIC-10, and TinyImageNet-200, FEAST achieves the highest population-averaged accuracy among the evaluated weight-sharing methods when each client receives its largest affordable subnetwork. Sub-supernet routing reduces aggregate model-parameter traffic by $6.8\times$ relative to full-supernet transmission.

Related papers

• FANS: Federated Adaptive Network Search Learning for Heterogeneous Devices [44.62284429272935]
  Heterogeneous Learning (HFL) aims to train models across devices with diverse resource budgets while preserving data privacy.<n>We introduce Federated Adaptive Network Search (FANS), a hypernetwork-based framework that learns a shared architecture space.<n>We evaluate FANS on CIFAR-10, CIFAR-100, and MNLI using ResNet-18, DenseNet-121, and BERT-base, respectively.
  arXiv  Detail & Related papers  (2026-09-05T13:57:56Z)
• Unextractable Protocol Models: Collaborative Training and Inference without Weight Materialization [58.14514930760722]
  We consider a decentralized setup in which the participants collaboratively train and serve a large neural network.<n>In this setup, we explore the possibility of unmaterializable weights, where a full weight set is never available to any one participant.<n>We introduce Unextractable Protocol Models (UPMs): a training and inference framework that leverages the sharded model setup.
  arXiv  Detail & Related papers  (2026-05-22T10:24:57Z)
• Learning to Compose SuperWeights for Neural Parameter Allocation Search [61.078949532440724]
  We show that our approach can generate parameters for many network using the same set of weights. This enables us to support tasks like efficient ensembling and anytime prediction.
  arXiv  Detail & Related papers  (2023-12-03T04:20:02Z)
• On the Convergence of Federated Averaging under Partial Participation for Over-parameterized Neural Networks [13.2844023993979]
  Federated learning (FL) is a widely employed distributed paradigm for collaboratively machine learning models from multiple clients without sharing local data. In this paper, we show that FedAvg converges to a global minimum at a global rate at a global focus.
  arXiv  Detail & Related papers  (2023-10-09T07:56:56Z)
• Mixture-of-Supernets: Improving Weight-Sharing Supernet Training with Architecture-Routed Mixture-of-Experts [55.470959564665705]
  Weight-sharing supernets are crucial for performance estimation in cutting-edge neural search frameworks. The proposed method attains state-of-the-art (SoTA) performance in NAS for fast machine translation models. It excels in NAS for building memory-efficient task-agnostic BERT models.
  arXiv  Detail & Related papers  (2023-06-08T00:35:36Z)
• SuperFedNAS: Cost-Efficient Federated Neural Architecture Search for On-Device Inference [7.860030408676541]
  Neural Architecture Search (NAS) for Federated Learning (FL) is an emerging field. It automates the design and training of Deep Neural Networks (DNNs) when data cannot be centralized due to privacy, communication costs, or regulatory restrictions. Recent federated NAS methods not only reduce manual effort but also help achieve higher accuracy than traditional FL methods like FedAvg.
  arXiv  Detail & Related papers  (2023-01-26T00:17:10Z)
• Optimizing Server-side Aggregation For Robust Federated Learning via Subspace Training [80.03567604524268]
  Non-IID data distribution across clients and poisoning attacks are two main challenges in real-world federated learning systems. We propose SmartFL, a generic approach that optimize the server-side aggregation process. We provide theoretical analyses of the convergence and generalization capacity for SmartFL.
  arXiv  Detail & Related papers  (2022-11-10T13:20:56Z)
• Sparse Random Networks for Communication-Efficient Federated Learning [23.614934319624826]
  One main challenge in federated learning is the large communication cost of exchanging weight updates from clients to the server at each round. We propose a radically different approach that does not update the weights at all. Instead, our method freezes the weights at their initial emphrandom values and learns how to sparsify the random network for the best performance.
  arXiv  Detail & Related papers  (2022-09-30T09:11:09Z)
• Supernet Training for Federated Image Classification under System Heterogeneity [15.2292571922932]
  In this work, we propose a novel framework to consider both scenarios, namely Federation of Supernet Training (FedSup) It is inspired by how averaging parameters in the model aggregation stage of Federated Learning (FL) is similar to weight-sharing in supernet training. Under our framework, we present an efficient algorithm (E-FedSup) by sending the sub-model to clients in the broadcast stage for reducing communication costs and training overhead.
  arXiv  Detail & Related papers  (2022-06-03T02:21:01Z)
• An Expectation-Maximization Perspective on Federated Learning [75.67515842938299]
  Federated learning describes the distributed training of models across multiple clients while keeping the data private on-device. In this work, we view the server-orchestrated federated learning process as a hierarchical latent variable model where the server provides the parameters of a prior distribution over the client-specific model parameters. We show that with simple Gaussian priors and a hard version of the well known Expectation-Maximization (EM) algorithm, learning in such a model corresponds to FedAvg, the most popular algorithm for the federated learning setting.
  arXiv  Detail & Related papers  (2021-11-19T12:58:59Z)
• ProgFed: Effective, Communication, and Computation Efficient Federated Learning by Progressive Training [65.68511423300812]
  We propose ProgFed, a progressive training framework for efficient and effective federated learning. ProgFed inherently reduces computation and two-way communication costs while maintaining the strong performance of the final models. Our results show that ProgFed converges at the same rate as standard training on full models.
  arXiv  Detail & Related papers  (2021-10-11T14:45:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.