ISFL: Federated Learning for Non-i.i.d. Data with Local Importance Sampling

• URL: http://arxiv.org/abs/2210.02119v3
• Date: Sun, 12 May 2024 05:49:38 GMT
• Title: ISFL: Federated Learning for Non-i.i.d. Data with Local Importance Sampling
• Authors: Zheqi Zhu, Yuchen Shi, Pingyi Fan, Chenghui Peng, Khaled B. Letaief,
• Abstract summary: We propose importance sampling federated learning (ISFL), an explicit framework with theoretical guarantees. We derive the convergence theorem of ISFL to involve the effects of local importance sampling. We employ a water-filling method to calculate the IS weights and develop the ISFL algorithms.
• Score: 17.29669920752378
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As a promising learning paradigm integrating computation and communication, federated learning (FL) proceeds the local training and the periodic sharing from distributed clients. Due to the non-i.i.d. data distribution on clients, FL model suffers from the gradient diversity, poor performance, bad convergence, etc. In this work, we aim to tackle this key issue by adopting importance sampling (IS) for local training. We propose importance sampling federated learning (ISFL), an explicit framework with theoretical guarantees. Firstly, we derive the convergence theorem of ISFL to involve the effects of local importance sampling. Then, we formulate the problem of selecting optimal IS weights and obtain the theoretical solutions. We also employ a water-filling method to calculate the IS weights and develop the ISFL algorithms. The experimental results on CIFAR-10 fit the proposed theorems well and verify that ISFL reaps better performance, sampling efficiency, as well as explainability on non-i.i.d. data. To the best of our knowledge, ISFL is the first non-i.i.d. FL solution from the local sampling aspect which exhibits theoretical compatibility with neural network models. Furthermore, as a local sampling approach, ISFL can be easily migrated into other emerging FL frameworks.

Related papers

• Over-the-Air Fair Federated Learning via Multi-Objective Optimization [52.295563400314094]
  We propose an over-the-air fair federated learning algorithm (OTA-FFL) to train fair FL models. Experiments demonstrate the superiority of OTA-FFL in achieving fairness and robust performance.
  arXiv  Detail & Related papers  (2025-01-06T21:16:51Z)
• SemiDFL: A Semi-Supervised Paradigm for Decentralized Federated Learning [12.542161138042632]
  Decentralized federated learning (DFL) realizes cooperative model training among connected clients without relying on a central server. Most existing work on DFL focuses on supervised learning, assuming each client possesses sufficient labeled data for local training. We propose SemiDFL, the first semi-supervised DFL method that enhances DFL performance in SSL scenarios by establishing a consensus in both data and model spaces.
  arXiv  Detail & Related papers  (2024-12-18T08:12:55Z)
• GAI-Enabled Explainable Personalized Federated Semi-Supervised Learning [29.931169585178818]
  Federated learning (FL) is a commonly distributed algorithm for mobile users (MUs) training artificial intelligence (AI) models. We propose an explainable personalized FL framework, called XPFL. Particularly, in local training, we utilize a generative AI (GAI) model to learn from large unlabeled data. In global aggregation, we obtain the new local local model by fusing the local and global FL models in specific proportions. Finally, simulation results validate the effectiveness of the proposed XPFL framework.
  arXiv  Detail & Related papers  (2024-10-11T08:58:05Z)
• R-SFLLM: Jamming Resilient Framework for Split Federated Learning with Large Language Models [83.77114091471822]
  Split federated learning (SFL) is a compute-efficient paradigm in distributed machine learning (ML) A challenge in SFL, particularly when deployed over wireless channels, is the susceptibility of transmitted model parameters to adversarial jamming. This is particularly pronounced for word embedding parameters in large language models (LLMs), which are crucial for language understanding. A physical layer framework is developed for resilient SFL with LLMs (R-SFLLM) over wireless networks.
  arXiv  Detail & Related papers  (2024-07-16T12:21:29Z)
• SpaFL: Communication-Efficient Federated Learning with Sparse Models and Low computational Overhead [75.87007729801304]
  SpaFL: a communication-efficient FL framework is proposed to optimize sparse model structures with low computational overhead. To optimize the pruning process itself, only thresholds are communicated between a server and clients instead of parameters. Global thresholds are used to update model parameters by extracting aggregated parameter importance.
  arXiv  Detail & Related papers  (2024-06-01T13:10:35Z)
• AdaptSFL: Adaptive Split Federated Learning in Resource-constrained Edge Networks [15.195798715517315]
  Split federated learning (SFL) is a promising solution by of floading the primary training workload to a server via model partitioning. We propose AdaptSFL, a novel resource-adaptive SFL framework, to expedite SFL under resource-constrained edge computing systems.
  arXiv  Detail & Related papers  (2024-03-19T19:05:24Z)
• Semi-Federated Learning: Convergence Analysis and Optimization of A Hybrid Learning Framework [70.83511997272457]
  We propose a semi-federated learning (SemiFL) paradigm to leverage both the base station (BS) and devices for a hybrid implementation of centralized learning (CL) and FL. We propose a two-stage algorithm to solve this intractable problem, in which we provide the closed-form solutions to the beamformers.
  arXiv  Detail & Related papers  (2023-10-04T03:32:39Z)
• Automated Federated Learning in Mobile Edge Networks -- Fast Adaptation and Convergence [83.58839320635956]
  Federated Learning (FL) can be used in mobile edge networks to train machine learning models in a distributed manner. Recent FL has been interpreted within a Model-Agnostic Meta-Learning (MAML) framework, which brings FL significant advantages in fast adaptation and convergence over heterogeneous datasets. This paper addresses how much benefit MAML brings to FL and how to maximize such benefit over mobile edge networks.
  arXiv  Detail & Related papers  (2023-03-23T02:42:10Z)
• Vertical Semi-Federated Learning for Efficient Online Advertising [50.18284051956359]
  Semi-VFL (Vertical Semi-Federated Learning) is proposed to achieve a practical industry application fashion for VFL. We build an inference-efficient single-party student model applicable to the whole sample space. New representation distillation methods are designed to extract cross-party feature correlations for both the overlapped and non-overlapped data.
  arXiv  Detail & Related papers  (2022-09-30T17:59:27Z)
• On the Importance and Applicability of Pre-Training for Federated Learning [28.238484580662785]
  We conduct a systematic study to explore pre-training for federated learning. We find that pre-training can improve FL, but also close its accuracy gap to the counterpart centralized learning. We conclude our paper with an attempt to understand the effect of pre-training on FL.
  arXiv  Detail & Related papers  (2022-06-23T06:02:33Z)
• TiFL: A Tier-based Federated Learning System [17.74678728280232]
  Federated Learning (FL) enables learning a shared model across many clients without violating the privacy requirements. We conduct a case study to show that heterogeneity in resource and data has a significant impact on training time and model accuracy in conventional FL systems. We propose TiFL, a Tier-based Federated Learning System, which divides clients into tiers based on their training performance and selects clients from the same tier in each training round.
  arXiv  Detail & Related papers  (2020-01-25T01:40:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.