Equitable-FL: Federated Learning with Sparsity for Resource-Constrained Environment

• URL: http://arxiv.org/abs/2309.00864v1
• Date: Sat, 2 Sep 2023 08:40:17 GMT
• Title: Equitable-FL: Federated Learning with Sparsity for Resource-Constrained Environment
• Authors: Indrajeet Kumar Sinha, Shekhar Verma, Krishna Pratap Singh
• Abstract summary: We propose a sparse form of federated learning that performs well in a Resource Constrained Environment. Our goal is to make learning possible, regardless of a node's space, computing, or bandwidth scarcity. Results obtained from experiments performed for training convolutional neural networks validate the efficacy of Equitable-FL.
• Score: 10.980548731600116
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In Federated Learning, model training is performed across multiple computing devices, where only parameters are shared with a common central server without exchanging their data instances. This strategy assumes abundance of resources on individual clients and utilizes these resources to build a richer model as user's models. However, when the assumption of the abundance of resources is violated, learning may not be possible as some nodes may not be able to participate in the process. In this paper, we propose a sparse form of federated learning that performs well in a Resource Constrained Environment. Our goal is to make learning possible, regardless of a node's space, computing, or bandwidth scarcity. The method is based on the observation that model size viz a viz available resources defines resource scarcity, which entails that reduction of the number of parameters without affecting accuracy is key to model training in a resource-constrained environment. In this work, the Lottery Ticket Hypothesis approach is utilized to progressively sparsify models to encourage nodes with resource scarcity to participate in collaborative training. We validate Equitable-FL on the $MNIST$, $F-MNIST$, and $CIFAR-10$ benchmark datasets, as well as the $Brain-MRI$ data and the $PlantVillage$ datasets. Further, we examine the effect of sparsity on performance, model size compaction, and speed-up for training. Results obtained from experiments performed for training convolutional neural networks validate the efficacy of Equitable-FL in heterogeneous resource-constrained learning environment.

Related papers

• A Resource-Adaptive Approach for Federated Learning under Resource-Constrained Environments [22.038826059430242]
  The paper studies a fundamental federated learning (FL) problem involving multiple clients with heterogeneous constrained resources. We propose Fed-RAA: a Resource-Adaptive Asynchronous Federated learning algorithm.
  arXiv  Detail & Related papers  (2024-06-19T08:55:40Z)
• subMFL: Compatiple subModel Generation for Federated Learning in Device Heterogenous Environment [0.2796197251957244]
  Federated Learning (FL) is commonly used in systems with distributed and heterogeneous devices. We propose a model compression approach that enables heterogeneous devices with varying computing capacities to participate in the FL process.
  arXiv  Detail & Related papers  (2024-05-30T12:49:34Z)
• FedHPL: Efficient Heterogeneous Federated Learning with Prompt Tuning and Logit Distillation [32.305134875959226]
  Federated learning (FL) is a privacy-preserving paradigm that enables distributed clients to collaboratively train models with a central server. We propose FedHPL, a parameter-efficient unified $textbfFed$erated learning framework for $textbfH$eterogeneous settings. We show that our framework outperforms state-of-the-art FL approaches, with less overhead and training rounds.
  arXiv  Detail & Related papers  (2024-05-27T15:25:32Z)
• Toward efficient resource utilization at edge nodes in federated learning [0.6990493129893112]
  Federated learning enables edge nodes to collaboratively contribute to constructing a global model without sharing their data. computational resource constraints and network communication can become a severe bottleneck for larger model sizes typical for deep learning applications. We propose and evaluate a FL strategy inspired by transfer learning in order to reduce resource utilization on devices.
  arXiv  Detail & Related papers  (2023-09-19T07:04:50Z)
• Adaptive Parameterization of Deep Learning Models for Federated Learning [85.82002651944254]
  Federated Learning offers a way to train deep neural networks in a distributed fashion. It incurs a communication overhead as the model parameters or gradients need to be exchanged regularly during training. In this paper, we propose to utilise parallel Adapters for Federated Learning.
  arXiv  Detail & Related papers  (2023-02-06T17:30:33Z)
• Scalable Collaborative Learning via Representation Sharing [53.047460465980144]
  Federated learning (FL) and Split Learning (SL) are two frameworks that enable collaborative learning while keeping the data private (on device) In FL, each data holder trains a model locally and releases it to a central server for aggregation. In SL, the clients must release individual cut-layer activations (smashed data) to the server and wait for its response (during both inference and back propagation). In this work, we present a novel approach for privacy-preserving machine learning, where the clients collaborate via online knowledge distillation using a contrastive loss.
  arXiv  Detail & Related papers  (2022-11-20T10:49:22Z)
• Network Aware Compute and Memory Allocation in Optically Composable Data Centres with Deep Reinforcement Learning and Graph Neural Networks [0.0]
  Resource-disaggregated data centre architectures promise a means of pooling resources remotely within data centres. We show how this can be done using an optically switched circuit circuit backbone in the data centre network (DCN) We show how emphdeep reinforcement learning can be used to learn effective emphnetwork-aware and emphtopologically-scalable allocation policies end-to-end.
  arXiv  Detail & Related papers  (2022-10-26T09:46:50Z)
• An Expectation-Maximization Perspective on Federated Learning [75.67515842938299]
  Federated learning describes the distributed training of models across multiple clients while keeping the data private on-device. In this work, we view the server-orchestrated federated learning process as a hierarchical latent variable model where the server provides the parameters of a prior distribution over the client-specific model parameters. We show that with simple Gaussian priors and a hard version of the well known Expectation-Maximization (EM) algorithm, learning in such a model corresponds to FedAvg, the most popular algorithm for the federated learning setting.
  arXiv  Detail & Related papers  (2021-11-19T12:58:59Z)
• LCS: Learning Compressible Subspaces for Adaptive Network Compression at Inference Time [57.52251547365967]
  We propose a method for training a "compressible subspace" of neural networks that contains a fine-grained spectrum of models. We present results for achieving arbitrarily fine-grained accuracy-efficiency trade-offs at inference time for structured and unstructured sparsity. Our algorithm extends to quantization at variable bit widths, achieving accuracy on par with individually trained networks.
  arXiv  Detail & Related papers  (2021-10-08T17:03:34Z)
• Clustered Federated Learning via Generalized Total Variation Minimization [83.26141667853057]
  We study optimization methods to train local (or personalized) models for local datasets with a decentralized network structure. Our main conceptual contribution is to formulate federated learning as total variation minimization (GTV) Our main algorithmic contribution is a fully decentralized federated learning algorithm.
  arXiv  Detail & Related papers  (2021-05-26T18:07:19Z)
• Blockchain Assisted Decentralized Federated Learning (BLADE-FL): Performance Analysis and Resource Allocation [119.19061102064497]
  We propose a decentralized FL framework by integrating blockchain into FL, namely, blockchain assisted decentralized federated learning (BLADE-FL) In a round of the proposed BLADE-FL, each client broadcasts its trained model to other clients, competes to generate a block based on the received models, and then aggregates the models from the generated block before its local training of the next round. We explore the impact of lazy clients on the learning performance of BLADE-FL, and characterize the relationship among the optimal K, the learning parameters, and the proportion of lazy clients.
  arXiv  Detail & Related papers  (2021-01-18T07:19:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.