Provably Near-Optimal Federated Ensemble Distillation with Negligible Overhead

• URL: http://arxiv.org/abs/2502.06349v2
• Date: Mon, 19 May 2025 08:10:04 GMT
• Title: Provably Near-Optimal Federated Ensemble Distillation with Negligible Overhead
• Abstract summary: Federated ensemble distillation addresses client heterogeneity by generating pseudo-labels for an unlabeled server dataset based on client predictions.<n>We propose a provably near-optimal weighting method that leverages client discriminators trained with a server-distributed generator and local datasets.<n>Our experiments on various image classification tasks demonstrate that the proposed method significantly outperforms baselines.
• Score: 11.115044922684545
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Federated ensemble distillation addresses client heterogeneity by generating pseudo-labels for an unlabeled server dataset based on client predictions and training the server model using the pseudo-labeled dataset. The unlabeled server dataset can either be pre-existing or generated through a data-free approach. The effectiveness of this approach critically depends on the method of assigning weights to client predictions when creating pseudo-labels, especially in highly heterogeneous settings. Inspired by theoretical results from GANs, we propose a provably near-optimal weighting method that leverages client discriminators trained with a server-distributed generator and local datasets. Our experiments on various image classification tasks demonstrate that the proposed method significantly outperforms baselines. Furthermore, we show that the additional communication cost, client-side privacy leakage, and client-side computational overhead introduced by our method are negligible, both in scenarios with and without a pre-existing server dataset.

Related papers

• FedCC: Towards Addressing Label Distribution Skews in Distillation-Based Federated Learning [6.693752118202639]
  Federated Learning (FL) enables distributed clients to collaboratively train models without sharing raw data.<n>In distillation-based FL, each client applies its local model on an unlabeled public dataset, and shares only prediction results with the server.<n>We propose FedCC, a simple and effective algorithm for mitigating client misclassification.
  arXiv  Detail & Related papers  (2026-08-24T09:34:26Z)
• One-Shot Hierarchical Federated Clustering [51.490181220883905]
  This paper introduces an efficient one-shot hierarchical Federated Clustering framework.<n>It performs client-end distribution exploration and server-end distribution aggregation.<n>It turns out that the complex cluster distributions across clients can be efficiently explored.
  arXiv  Detail & Related papers  (2026-01-10T02:58:33Z)
• CATCHFed: Efficient Unlabeled Data Utilization for Semi-Supervised Federated Learning in Limited Labels Environments [3.2805451821208234]
  Federated learning is a promising paradigm that utilizes distributed client resources while preserving data privacy.<n>Most existing FL approaches assume clients possess labeled data, however, in real-world scenarios, client-side labels are often unavailable.<n>We propose textitCATCHFed, which introduces client-aware adaptive thresholds considering class difficulty, hybrid thresholds to enhance pseudo-label quality, and utilizes unpseudo-labeled data for consistency regularization.
  arXiv  Detail & Related papers  (2025-11-14T13:16:52Z)
• Towards Federated Clustering: A Client-wise Private Graph Aggregation Framework [57.04850867402913]
  Federated clustering addresses the challenge of extracting patterns from decentralized, unlabeled data.<n>We propose Structural Privacy-Preserving Federated Graph Clustering (SPP-FGC), a novel algorithm that innovatively leverages local structural graphs as the primary medium for privacy-preserving knowledge sharing.<n>Our framework achieves state-of-the-art performance, improving clustering accuracy by up to 10% (NMI) over federated baselines while maintaining provable privacy guarantees.
  arXiv  Detail & Related papers  (2025-11-14T03:05:22Z)
• Uncertainty-Aware Label Refinement on Hypergraphs for Personalized Federated Facial Expression Recognition [58.98052764581606]
  We develop a novel uncertainty-Aware label refineMent on hYpergraphs (AMY) method.<n>For local training, each local model consists of a backbone, an uncertainty estimation (UE) block, and an expression classification (EC) block.<n>A personalized uncertainty estimator is then introduced to estimate reliable uncertainty weights of samples in the local client.
  arXiv  Detail & Related papers  (2025-01-03T13:59:21Z)
• Boosting Federated Learning with FedEntOpt: Mitigating Label Skew by Entropy-Based Client Selection [13.851391819710367]
  Deep learning domains typically require an extensive amount of data for optimal performance.<n>FedEntOpt is designed to mitigate performance issues caused by label distribution skew.<n>It exhibits robust and superior performance in scenarios with low participation rates and client dropout.
  arXiv  Detail & Related papers  (2024-11-02T13:31:36Z)
• Learning Unlabeled Clients Divergence for Federated Semi-Supervised Learning via Anchor Model Aggregation [10.282711631100845]
  SemiAnAgg learns unlabeled client contributions via an anchor model. SemiAnAgg achieves new state-of-the-art results on four widely used FedSemi benchmarks.
  arXiv  Detail & Related papers  (2024-07-14T20:50:40Z)
• Provably Personalized and Robust Federated Learning [47.50663360022456]
  We propose simple algorithms which identify clusters of similar clients and train a personalized modelper-cluster. The convergence rates of our algorithmsally match those obtained if we knew the true underlying clustering of the clients and are provably robust in the Byzantine setting.
  arXiv  Detail & Related papers  (2023-06-14T09:37:39Z)
• Exploring One-shot Semi-supervised Federated Learning with A Pre-trained Diffusion Model [40.83058938096914]
  We propose FedDISC, a Federated Diffusion-Inspired Semi-supervised Co-training method. We first extract prototypes of the labeled server data and use these prototypes to predict pseudo-labels of the client data. For each category, we compute the cluster centroids and domain-specific representations to signify the semantic and stylistic information of their distributions. These representations are sent back to the server, which uses the pre-trained to generate synthetic datasets complying with the client distributions and train a global model on it.
  arXiv  Detail & Related papers  (2023-05-06T14:22:33Z)
• FedIL: Federated Incremental Learning from Decentralized Unlabeled Data with Convergence Analysis [23.70951896315126]
  This work considers the server with a small labeled dataset and intends to use unlabeled data in multiple clients for semi-supervised learning. We propose a new framework with a generalized model, Federated Incremental Learning (FedIL), to address the problem of how to utilize labeled data in the server and unlabeled data in clients separately.
  arXiv  Detail & Related papers  (2023-02-23T07:12:12Z)
• When to Trust Aggregated Gradients: Addressing Negative Client Sampling in Federated Learning [41.51682329500003]
  We propose a novel learning rate adaptation mechanism to adjust the server learning rate for the aggregated gradient in each round. We make theoretical deductions to find a meaningful and robust indicator that is positively related to the optimal server learning rate.
  arXiv  Detail & Related papers  (2023-01-25T03:52:45Z)
• Rethinking Data Heterogeneity in Federated Learning: Introducing a New Notion and Standard Benchmarks [65.34113135080105]
  We show that not only the issue of data heterogeneity in current setups is not necessarily a problem but also in fact it can be beneficial for the FL participants. Our observations are intuitive. Our code is available at https://github.com/MMorafah/FL-SC-NIID.
  arXiv  Detail & Related papers  (2022-09-30T17:15:19Z)
• Fed-CBS: A Heterogeneity-Aware Client Sampling Mechanism for Federated Learning via Class-Imbalance Reduction [76.26710990597498]
  We show that the class-imbalance of the grouped data from randomly selected clients can lead to significant performance degradation. Based on our key observation, we design an efficient client sampling mechanism, i.e., Federated Class-balanced Sampling (Fed-CBS) In particular, we propose a measure of class-imbalance and then employ homomorphic encryption to derive this measure in a privacy-preserving way.
  arXiv  Detail & Related papers  (2022-09-30T05:42:56Z)
• Straggler-Resilient Personalized Federated Learning [55.54344312542944]
  Federated learning allows training models from samples distributed across a large network of clients while respecting privacy and communication restrictions. We develop a novel algorithmic procedure with theoretical speedup guarantees that simultaneously handles two of these hurdles. Our method relies on ideas from representation learning theory to find a global common representation using all clients' data and learn a user-specific set of parameters leading to a personalized solution for each client.
  arXiv  Detail & Related papers  (2022-06-05T01:14:46Z)
• Dash: Semi-Supervised Learning with Dynamic Thresholding [72.74339790209531]
  We propose a semi-supervised learning (SSL) approach that uses unlabeled examples to train models. Our proposed approach, Dash, enjoys its adaptivity in terms of unlabeled data selection.
  arXiv  Detail & Related papers  (2021-09-01T23:52:29Z)
• Federated Multi-Target Domain Adaptation [99.93375364579484]
  Federated learning methods enable us to train machine learning models on distributed user data while preserving its privacy. We consider a more practical scenario where the distributed client data is unlabeled, and a centralized labeled dataset is available on the server. We propose an effective DualAdapt method to address the new challenges.
  arXiv  Detail & Related papers  (2021-08-17T17:53:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.