DYNAFED: Tackling Client Data Heterogeneity with Global Dynamics

• URL: http://arxiv.org/abs/2211.10878v1
• Date: Sun, 20 Nov 2022 06:13:06 GMT
• Title: DYNAFED: Tackling Client Data Heterogeneity with Global Dynamics
• Authors: Renjie Pi, Weizhong Zhang, Yueqi Xie, Jiahui Gao, Xiaoyu Wang, Sunghun Kim, Qifeng Chen
• Abstract summary: Local training on non-iid distributed data results in deflected local optimum. A natural solution is to gather all client data onto the server, such that the server has a global view of the entire data distribution. In this paper, we put forth an idea to collect and leverage global knowledge on the server without hindering data privacy.
• Score: 60.60173139258481
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The Federated Learning (FL) paradigm is known to face challenges under heterogeneous client data. Local training on non-iid distributed data results in deflected local optimum, which causes the client models drift further away from each other and degrades the aggregated global model's performance. A natural solution is to gather all client data onto the server, such that the server has a global view of the entire data distribution. Unfortunately, this reduces to regular training, which compromises clients' privacy and conflicts with the purpose of FL. In this paper, we put forth an idea to collect and leverage global knowledge on the server without hindering data privacy. We unearth such knowledge from the dynamics of the global model's trajectory. Specifically, we first reserve a short trajectory of global model snapshots on the server. Then, we synthesize a small pseudo dataset such that the model trained on it mimics the dynamics of the reserved global model trajectory. Afterward, the synthesized data is used to help aggregate the deflected clients into the global model. We name our method Dynafed, which enjoys the following advantages: 1) we do not rely on any external on-server dataset, which requires no additional cost for data collection; 2) the pseudo data can be synthesized in early communication rounds, which enables Dynafed to take effect early for boosting the convergence and stabilizing training; 3) the pseudo data only needs to be synthesized once and can be directly utilized on the server to help aggregation in subsequent rounds. Experiments across extensive benchmarks are conducted to showcase the effectiveness of Dynafed. We also provide insights and understanding of the underlying mechanism of our method.

Related papers

• ConDa: Fast Federated Unlearning with Contribution Dampening [46.074452659791575]
  ConDa is a framework that performs efficient unlearning by tracking down the parameters which affect the global model for each client. We perform experiments on multiple datasets and demonstrate that ConDa is effective to forget a client's data.
  arXiv  Detail & Related papers  (2024-10-05T12:45:35Z)
• An Aggregation-Free Federated Learning for Tackling Data Heterogeneity [50.44021981013037]
  Federated Learning (FL) relies on the effectiveness of utilizing knowledge from distributed datasets. Traditional FL methods adopt an aggregate-then-adapt framework, where clients update local models based on a global model aggregated by the server from the previous training round. We introduce FedAF, a novel aggregation-free FL algorithm.
  arXiv  Detail & Related papers  (2024-04-29T05:55:23Z)
• LEFL: Low Entropy Client Sampling in Federated Learning [6.436397118145477]
  Federated learning (FL) is a machine learning paradigm where multiple clients collaborate to optimize a single global model using their private data. We propose LEFL, an alternative sampling strategy by performing a one-time clustering of clients based on their model's learned high-level features. We show of sampled clients selected with this approach yield a low relative entropy with respect to the global data distribution.
  arXiv  Detail & Related papers  (2023-12-29T01:44:20Z)
• Federated Learning with Projected Trajectory Regularization [65.6266768678291]
  Federated learning enables joint training of machine learning models from distributed clients without sharing their local data. One key challenge in federated learning is to handle non-identically distributed data across the clients. We propose a novel federated learning framework with projected trajectory regularization (FedPTR) for tackling the data issue.
  arXiv  Detail & Related papers  (2023-12-22T02:12:08Z)
• Towards Instance-adaptive Inference for Federated Learning [80.38701896056828]
  Federated learning (FL) is a distributed learning paradigm that enables multiple clients to learn a powerful global model by aggregating local training. In this paper, we present a novel FL algorithm, i.e., FedIns, to handle intra-client data heterogeneity by enabling instance-adaptive inference in the FL framework. Our experiments show that our FedIns outperforms state-of-the-art FL algorithms, e.g., a 6.64% improvement against the top-performing method with less than 15% communication cost on Tiny-ImageNet.
  arXiv  Detail & Related papers  (2023-08-11T09:58:47Z)
• Federated Learning for Semantic Parsing: Task Formulation, Evaluation Setup, New Algorithms [29.636944156801327]
  Multiple clients collaboratively train one global model without sharing their semantic parsing data. Lorar adjusts each client's contribution to the global model update based on its training loss reduction during each round. Clients with smaller datasets enjoy larger performance gains.
  arXiv  Detail & Related papers  (2023-05-26T19:25:49Z)
• Towards Fair Federated Learning with Zero-Shot Data Augmentation [123.37082242750866]
  Federated learning has emerged as an important distributed learning paradigm, where a server aggregates a global model from many client-trained models while having no access to the client data. We propose a novel federated learning system that employs zero-shot data augmentation on under-represented data to mitigate statistical heterogeneity and encourage more uniform accuracy performance across clients in federated networks. We study two variants of this scheme, Fed-ZDAC (federated learning with zero-shot data augmentation at the clients) and Fed-ZDAS (federated learning with zero-shot data augmentation at the server).
  arXiv  Detail & Related papers  (2021-04-27T18:23:54Z)
• FedProf: Optimizing Federated Learning with Dynamic Data Profiling [9.74942069718191]
  Federated Learning (FL) has shown great potential as a privacy-preserving solution to learning from decentralized data. A large proportion of the clients are probably in possession of only low-quality data that are biased, noisy or even irrelevant. We propose a novel approach to optimizing FL under such circumstances without breaching data privacy.
  arXiv  Detail & Related papers  (2021-02-02T20:10:14Z)
• Toward Understanding the Influence of Individual Clients in Federated Learning [52.07734799278535]
  Federated learning allows clients to jointly train a global model without sending their private data to a central server. We defined a new notion called em-Influence, quantify this influence over parameters, and proposed an effective efficient model to estimate this metric.
  arXiv  Detail & Related papers  (2020-12-20T14:34:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.