Shuffled Model of Federated Learning: Privacy, Communication and Accuracy Trade-offs

• URL: http://arxiv.org/abs/2008.07180v2
• Date: Wed, 23 Sep 2020 16:54:50 GMT
• Title: Shuffled Model of Federated Learning: Privacy, Communication and Accuracy Trade-offs
• Authors: Antonious M. Girgis, Deepesh Data, Suhas Diggavi, Peter Kairouz, and Ananda Theertha Suresh
• Abstract summary: We consider a distributed empirical risk minimization (ERM) optimization problem with communication efficiency and privacy requirements. We develop (optimal) communication-efficient schemes for private mean estimation for several $ell_p$ spaces. We demonstrate that one can get the same privacy, optimization-performance operating point developed in recent methods that use full-precision communication.
• Score: 30.58690911428577
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We consider a distributed empirical risk minimization (ERM) optimization problem with communication efficiency and privacy requirements, motivated by the federated learning (FL) framework. Unique challenges to the traditional ERM problem in the context of FL include (i) need to provide privacy guarantees on clients' data, (ii) compress the communication between clients and the server, since clients might have low-bandwidth links, (iii) work with a dynamic client population at each round of communication between the server and the clients, as a small fraction of clients are sampled at each round. To address these challenges we develop (optimal) communication-efficient schemes for private mean estimation for several $\ell_p$ spaces, enabling efficient gradient aggregation for each iteration of the optimization solution of the ERM. We also provide lower and upper bounds for mean estimation with privacy and communication constraints for arbitrary $\ell_p$ spaces. To get the overall communication, privacy, and optimization performance operation point, we combine this with privacy amplification opportunities inherent to this setup. Our solution takes advantage of the inherent privacy amplification provided by client sampling and data sampling at each client (through Stochastic Gradient Descent) as well as the recently developed privacy framework using anonymization, which effectively presents to the server responses that are randomly shuffled with respect to the clients. Putting these together, we demonstrate that one can get the same privacy, optimization-performance operating point developed in recent methods that use full-precision communication, but at a much lower communication cost, i.e., effectively getting communication efficiency for "free".

Related papers

• Using Synthetic Data to Mitigate Unfairness and Preserve Privacy through Single-Shot Federated Learning [6.516872951510096]
  We propose a strategy that promotes fair predictions across clients without the need to pass information between the clients and server. We then pass each client's synthetic dataset to the server, the collection of which is used to train the server model.
  arXiv  Detail & Related papers  (2024-09-14T21:04:11Z)
• Emulating Full Client Participation: A Long-Term Client Selection Strategy for Federated Learning [48.94952630292219]
  We propose a novel client selection strategy designed to emulate the performance achieved with full client participation. In a single round, we select clients by minimizing the gradient-space estimation error between the client subset and the full client set. In multi-round selection, we introduce a novel individual fairness constraint, which ensures that clients with similar data distributions have similar frequencies of being selected.
  arXiv  Detail & Related papers  (2024-05-22T12:27:24Z)
• Greedy Shapley Client Selection for Communication-Efficient Federated Learning [32.38170282930876]
  Standard client selection algorithms for Federated Learning (FL) are often unbiased and involve uniform random sampling of clients. We develop a biased client selection strategy, GreedyFed, that identifies and greedily selects the most contributing clients in each communication round. Compared to various client selection strategies on several real-world datasets, GreedyFed demonstrates fast and stable convergence with high accuracy under timing constraints.
  arXiv  Detail & Related papers  (2023-12-14T16:44:38Z)
• User-Centric Federated Learning: Trading off Wireless Resources for Personalization [18.38078866145659]
  In Federated Learning (FL) systems, Statistical Heterogeneousness increases the algorithm convergence time and reduces the generalization performance. To tackle the above problems without violating the privacy constraints that FL imposes, personalized FL methods have to couple statistically similar clients without directly accessing their data. In this work, we design user-centric aggregation rules that are based on readily available gradient information and are capable of producing personalized models for each FL client. Our algorithm outperforms popular personalized FL baselines in terms of average accuracy, worst node performance, and training communication overhead.
  arXiv  Detail & Related papers  (2023-04-25T15:45:37Z)
• Re-Weighted Softmax Cross-Entropy to Control Forgetting in Federated Learning [14.196701066823499]
  In Federated Learning, a global model is learned by aggregating model updates computed at a set of independent client nodes. We show that individual client models experience a catastrophic forgetting with respect to data from other clients. We propose an efficient approach that modifies the cross-entropy objective on a per-client basis by re-weighting the softmax logits prior to computing the loss.
  arXiv  Detail & Related papers  (2023-04-11T14:51:55Z)
• Regulating Clients' Noise Adding in Federated Learning without Verification [24.196751469021848]
  In federated learning, clients cooperatively train a global model without revealing their raw data but gradients or parameters. With such privacy concerns, a client may overly add artificial noise to his local updates to compromise the global model training. This paper proposes a novel pricing mechanism to regulate privacy-sensitive clients without verifying their parameter updates.
  arXiv  Detail & Related papers  (2023-02-24T16:44:15Z)
• FedFM: Anchor-based Feature Matching for Data Heterogeneity in Federated Learning [91.74206675452888]
  We propose a novel method FedFM, which guides each client's features to match shared category-wise anchors. To achieve higher efficiency and flexibility, we propose a FedFM variant, called FedFM-Lite, where clients communicate with server with fewer synchronization times and communication bandwidth costs.
  arXiv  Detail & Related papers  (2022-10-14T08:11:34Z)
• Improving Privacy-Preserving Vertical Federated Learning by Efficient Communication with ADMM [62.62684911017472]
  Federated learning (FL) enables devices to jointly train shared models while keeping the training data local for privacy purposes. We introduce a VFL framework with multiple heads (VIM), which takes the separate contribution of each client into account. VIM achieves significantly higher performance and faster convergence compared with the state-of-the-art.
  arXiv  Detail & Related papers  (2022-07-20T23:14:33Z)
• DisPFL: Towards Communication-Efficient Personalized Federated Learning via Decentralized Sparse Training [84.81043932706375]
  We propose a novel personalized federated learning framework in a decentralized (peer-to-peer) communication protocol named Dis-PFL. Dis-PFL employs personalized sparse masks to customize sparse local models on the edge. We demonstrate that our method can easily adapt to heterogeneous local clients with varying computation complexities.
  arXiv  Detail & Related papers  (2022-06-01T02:20:57Z)
• Low-Latency Federated Learning over Wireless Channels with Differential Privacy [142.5983499872664]
  In federated learning (FL), model training is distributed over clients and local models are aggregated by a central server. In this paper, we aim to minimize FL training delay over wireless channels, constrained by overall training performance as well as each client's differential privacy (DP) requirement.
  arXiv  Detail & Related papers  (2021-06-20T13:51:18Z)
• Faster Non-Convex Federated Learning via Global and Local Momentum [57.52663209739171]
  textttFedGLOMO is the first (first-order) FLtexttFedGLOMO algorithm. Our algorithm is provably optimal even with communication between the clients and the server.
  arXiv  Detail & Related papers  (2020-12-07T21:05:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.