Optimal Importance Sampling for Federated Learning

• URL: http://arxiv.org/abs/2010.13600v1
• Date: Mon, 26 Oct 2020 14:15:33 GMT
• Title: Optimal Importance Sampling for Federated Learning
• Authors: Elsa Rizk, Stefan Vlaski, Ali H. Sayed
• Abstract summary: Federated learning involves a mixture of centralized and decentralized processing tasks. The sampling of both agents and data is generally uniform; however, in this work we consider non-uniform sampling. We derive optimal importance sampling strategies for both agent and data selection and show that non-uniform sampling without replacement improves the performance of the original FedAvg algorithm.
• Score: 57.14673504239551
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Federated learning involves a mixture of centralized and decentralized processing tasks, where a server regularly selects a sample of the agents and these in turn sample their local data to compute stochastic gradients for their learning updates. This process runs continually. The sampling of both agents and data is generally uniform; however, in this work we consider non-uniform sampling. We derive optimal importance sampling strategies for both agent and data selection and show that non-uniform sampling without replacement improves the performance of the original FedAvg algorithm. We run experiments on a regression and classification problem to illustrate the theoretical results.

Related papers

• Balanced Data Sampling for Language Model Training with Clustering [96.46042695333655]
  We propose ClusterClip Sampling to balance the text distribution of training data for better model training. Extensive experiments validate the effectiveness of ClusterClip Sampling.
  arXiv  Detail & Related papers  (2024-02-22T13:20:53Z)
• Simple and effective data augmentation for compositional generalization [64.00420578048855]
  We show that data augmentation methods that sample MRs and backtranslate them can be effective for compositional generalization. Remarkably, sampling from a uniform distribution performs almost as well as sampling from the test distribution.
  arXiv  Detail & Related papers  (2024-01-18T09:13:59Z)
• Data Pruning via Moving-one-Sample-out [61.45441981346064]
  We propose a novel data-pruning approach called moving-one-sample-out (MoSo) MoSo aims to identify and remove the least informative samples from the training set. Experimental results demonstrate that MoSo effectively mitigates severe performance degradation at high pruning ratios.
  arXiv  Detail & Related papers  (2023-10-23T08:00:03Z)
• When to Trust Aggregated Gradients: Addressing Negative Client Sampling in Federated Learning [41.51682329500003]
  We propose a novel learning rate adaptation mechanism to adjust the server learning rate for the aggregated gradient in each round. We make theoretical deductions to find a meaningful and robust indicator that is positively related to the optimal server learning rate.
  arXiv  Detail & Related papers  (2023-01-25T03:52:45Z)
• Sampling Through the Lens of Sequential Decision Making [9.101505546901999]
  We propose a reward-guided sampling strategy called Adaptive Sample with Reward (ASR) Our approach optimally adjusts the sampling process to achieve optimal performance. Empirical results in information retrieval and clustering demonstrate ASR's superb performance across different datasets.
  arXiv  Detail & Related papers  (2022-08-17T04:01:29Z)
• Achieving Representative Data via Convex Hull Feasibility Sampling Algorithms [35.29582673348303]
  Sampling biases in training data are a major source of algorithmic biases in machine learning systems. We present adaptive sampling methods to determine, with high confidence, whether it is possible to assemble a representative dataset from the given data sources.
  arXiv  Detail & Related papers  (2022-04-13T23:14:05Z)
• Federated Learning under Importance Sampling [49.17137296715029]
  We study the effect of importance sampling and devise schemes for sampling agents and data non-uniformly guided by a performance measure. We find that in schemes involving sampling without replacement, the performance of the resulting architecture is controlled by two factors related to data variability at each agent.
  arXiv  Detail & Related papers  (2020-12-14T10:08:55Z)
• Robust Sampling in Deep Learning [62.997667081978825]
  Deep learning requires regularization mechanisms to reduce overfitting and improve generalization. We address this problem by a new regularization method based on distributional robust optimization. During the training, the selection of samples is done according to their accuracy in such a way that the worst performed samples are the ones that contribute the most in the optimization.
  arXiv  Detail & Related papers  (2020-06-04T09:46:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.