FedEMA-Distill: Exponential Moving Average Guided Knowledge Distillation for Robust Federated Learning

• URL: http://arxiv.org/abs/2603.04422v1
• Date: Sun, 15 Feb 2026 01:40:45 GMT
• Title: FedEMA-Distill: Exponential Moving Average Guided Knowledge Distillation for Robust Federated Learning
• Abstract summary: Federated learning (FL) often degrades when clients hold heterogeneous non-Independent and Identically Distributed (non-IID) data.<n>This paper proposes FedEMA-Distill, a server-side procedure that combines an exponential moving average (EMA) of the global model with ensemble knowledge distillation from client-uploaded prediction logits.
• Score: 1.1393603788068776
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Federated learning (FL) often degrades when clients hold heterogeneous non-Independent and Identically Distributed (non-IID) data and when some clients behave adversarially, leading to client drift, slow convergence, and high communication overhead. This paper proposes FedEMA-Distill, a server-side procedure that combines an exponential moving average (EMA) of the global model with ensemble knowledge distillation from client-uploaded prediction logits evaluated on a small public proxy dataset. Clients run standard local training, upload only compressed logits, and may use different model architectures, so no changes are required to client-side software while still supporting model heterogeneity across devices. Experiments on CIFAR-10, CIFAR-100, FEMNIST, and AG News under Dirichlet-0.1 label skew show that FedEMA-Distill improves top-1 accuracy by several percentage points (up to +5% on CIFAR-10 and +6% on CIFAR-100) over representative baselines, reaches a given target accuracy in 30-35% fewer communication rounds, and reduces per-round client uplink payloads to 0.09-0.46 MB, i.e., roughly an order of magnitude less than transmitting full model weights. Using coordinate-wise median or trimmed-mean aggregation of logits at the server further stabilizes training in the presence of up to 10-20% Byzantine clients and yields well-calibrated predictions under attack. These results indicate that coupling temporal smoothing with logits-only aggregation provides a communication-efficient and attack-resilient FL pipeline that is deployment-friendly and compatible with secure aggregation and differential privacy, since only aggregated or obfuscated model outputs are exchanged.

Related papers

• Optimized Federated Knowledge Distillation with Distributed Neural Architecture Search [7.461859467262201]
  Federated Learning (FL) enables collaborative model training without centralizing data.<n>Existing FL approaches mitigate these challenges through improved aggregation, personalization, or knowledge distillation.<n>This work introduces FedKDNAS, a distillation-driven FL framework that combines client-side neural architecture selection with distillation of server-coordinated knowledge.
  arXiv  Detail & Related papers  (2026-05-20T15:50:49Z)
• Prediction-space knowledge markets for communication-efficient federated learning on multimedia tasks [0.0]
  Federated learning (FL) enables collaborative training over distributed multimedia data.<n>We propose KTA v2, a prediction-space knowledge trading market for FL.<n>Experiments on FEMNIST, CIFAR-10 and AG News show that KTA v2 consistently outperforms a local-only baseline and strong parameter-based methods.
  arXiv  Detail & Related papers  (2025-11-30T11:24:19Z)
• FedSkipTwin: Digital-Twin-Guided Client Skipping for Communication-Efficient Federated Learning [2.209918162717035]
  Communication overhead remains a primary bottleneck in federated learning (FL)<n>This work introduces FedSkipTwin, a novel client-skipping algorithm driven by lightweight, server-side digital twins.<n>Experiments are conducted on the UCI-HAR and MNIST datasets with 10 clients under a non-IID data distribution.
  arXiv  Detail & Related papers  (2025-07-18T03:39:08Z)
• FedAR: Addressing Client Unavailability in Federated Learning with Local Update Approximation and Rectification [8.747592727421596]
  Federated learning (FL) enables clients to collaboratively train machine learning models under the coordination of a server. FedAR can get all clients involved in the global model update to achieve a high-quality global model on the server. FedAR also depicts impressive performance in the presence of a large number of clients with severe client unavailability.
  arXiv  Detail & Related papers  (2024-07-26T21:56:52Z)
• An Aggregation-Free Federated Learning for Tackling Data Heterogeneity [50.44021981013037]
  Federated Learning (FL) relies on the effectiveness of utilizing knowledge from distributed datasets. Traditional FL methods adopt an aggregate-then-adapt framework, where clients update local models based on a global model aggregated by the server from the previous training round. We introduce FedAF, a novel aggregation-free FL algorithm.
  arXiv  Detail & Related papers  (2024-04-29T05:55:23Z)
• FedCode: Communication-Efficient Federated Learning via Transferring Codebooks [3.004066195320147]
  Federated Learning (FL) is a distributed machine learning paradigm that enables learning models from decentralized local data. Existing approaches rely on model compression techniques, such as pruning and weight clustering to tackle this. We propose FedCode where clients transmit only codebooks, i.e., the cluster centers of updated model weight values.
  arXiv  Detail & Related papers  (2023-11-15T12:06:32Z)
• Client-side Gradient Inversion Against Federated Learning from Poisoning [59.74484221875662]
  Federated Learning (FL) enables distributed participants to train a global model without sharing data directly to a central server. Recent studies have revealed that FL is vulnerable to gradient inversion attack (GIA), which aims to reconstruct the original training samples. We propose Client-side poisoning Gradient Inversion (CGI), which is a novel attack method that can be launched from clients.
  arXiv  Detail & Related papers  (2023-09-14T03:48:27Z)
• Towards Instance-adaptive Inference for Federated Learning [80.38701896056828]
  Federated learning (FL) is a distributed learning paradigm that enables multiple clients to learn a powerful global model by aggregating local training. In this paper, we present a novel FL algorithm, i.e., FedIns, to handle intra-client data heterogeneity by enabling instance-adaptive inference in the FL framework. Our experiments show that our FedIns outperforms state-of-the-art FL algorithms, e.g., a 6.64% improvement against the top-performing method with less than 15% communication cost on Tiny-ImageNet.
  arXiv  Detail & Related papers  (2023-08-11T09:58:47Z)
• FedCliP: Federated Learning with Client Pruning [3.796320380104124]
  Federated learning (FL) is a newly emerging distributed learning paradigm. One fundamental bottleneck in FL is the heavy communication overheads between the distributed clients and the central server. We propose FedCliP, the first communication efficient FL training framework from a macro perspective.
  arXiv  Detail & Related papers  (2023-01-17T09:15:37Z)
• Acceleration of Federated Learning with Alleviated Forgetting in Local Training [61.231021417674235]
  Federated learning (FL) enables distributed optimization of machine learning models while protecting privacy. We propose FedReg, an algorithm to accelerate FL with alleviated knowledge forgetting in the local training stage. Our experiments demonstrate that FedReg not only significantly improves the convergence rate of FL, especially when the neural network architecture is deep.
  arXiv  Detail & Related papers  (2022-03-05T02:31:32Z)
• Towards Fair Federated Learning with Zero-Shot Data Augmentation [123.37082242750866]
  Federated learning has emerged as an important distributed learning paradigm, where a server aggregates a global model from many client-trained models while having no access to the client data. We propose a novel federated learning system that employs zero-shot data augmentation on under-represented data to mitigate statistical heterogeneity and encourage more uniform accuracy performance across clients in federated networks. We study two variants of this scheme, Fed-ZDAC (federated learning with zero-shot data augmentation at the clients) and Fed-ZDAS (federated learning with zero-shot data augmentation at the server).
  arXiv  Detail & Related papers  (2021-04-27T18:23:54Z)
• A Bayesian Federated Learning Framework with Online Laplace Approximation [144.7345013348257]
  Federated learning allows multiple clients to collaboratively learn a globally shared model. We propose a novel FL framework that uses online Laplace approximation to approximate posteriors on both the client and server side. We achieve state-of-the-art results on several benchmarks, clearly demonstrating the advantages of the proposed method.
  arXiv  Detail & Related papers  (2021-02-03T08:36:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.