FedDCT: Federated Learning of Large Convolutional Neural Networks on Resource Constrained Devices using Divide and Collaborative Training

• URL: http://arxiv.org/abs/2211.10948v2
• Date: Mon, 18 Sep 2023 15:21:25 GMT
• Title: FedDCT: Federated Learning of Large Convolutional Neural Networks on Resource Constrained Devices using Divide and Collaborative Training
• Authors: Quan Nguyen, Hieu H. Pham, Kok-Seng Wong, Phi Le Nguyen, Truong Thao Nguyen, Minh N. Do
• Abstract summary: We introduce FedDCT, a novel distributed learning paradigm that enables the usage of large, high-performance CNNs on resource-limited edge devices. We empirically conduct experiments on standardized datasets, including CIFAR-10, CIFAR-100, and two real-world medical datasets HAM10000 and VAIPE. Compared to other existing approaches, FedDCT achieves higher accuracy and substantially reduces the number of communication rounds.
• Score: 13.072061144206097
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce FedDCT, a novel distributed learning paradigm that enables the usage of large, high-performance CNNs on resource-limited edge devices. As opposed to traditional FL approaches, which require each client to train the full-size neural network independently during each training round, the proposed FedDCT allows a cluster of several clients to collaboratively train a large deep learning model by dividing it into an ensemble of several small sub-models and train them on multiple devices in parallel while maintaining privacy. In this collaborative training process, clients from the same cluster can also learn from each other, further improving their ensemble performance. In the aggregation stage, the server takes a weighted average of all the ensemble models trained by all the clusters. FedDCT reduces the memory requirements and allows low-end devices to participate in FL. We empirically conduct extensive experiments on standardized datasets, including CIFAR-10, CIFAR-100, and two real-world medical datasets HAM10000 and VAIPE. Experimental results show that FedDCT outperforms a set of current SOTA FL methods with interesting convergence behaviors. Furthermore, compared to other existing approaches, FedDCT achieves higher accuracy and substantially reduces the number of communication rounds (with $4-8$ times fewer memory requirements) to achieve the desired accuracy on the testing dataset without incurring any extra training cost on the server side.

Related papers

• Embracing Federated Learning: Enabling Weak Client Participation via Partial Model Training [21.89214794178211]
  In Federated Learning (FL), clients may have weak devices that cannot train the full model or even hold it in their memory space. We propose EmbracingFL, a general FL framework that allows all available clients to join the distributed training. Our empirical study shows that EmbracingFL consistently achieves high accuracy as like all clients are strong, outperforming the state-of-the-art width reduction methods.
  arXiv  Detail & Related papers  (2024-06-21T13:19:29Z)
• Harnessing Increased Client Participation with Cohort-Parallel Federated Learning [2.9593087583214173]
  Federated Learning (FL) is a machine learning approach where nodes collaboratively train a global model. We introduce Cohort-Parallel Federated Learning (CPFL), a novel learning approach where each cohort independently trains a global model. CPFL with four cohorts, non-IID data distribution, and CIFAR-10 yields a 1.9$times$ reduction in train time and a 1.3$times$ reduction in resource usage.
  arXiv  Detail & Related papers  (2024-05-24T15:34:09Z)
• Communication Efficient ConFederated Learning: An Event-Triggered SAGA Approach [67.27031215756121]
  Federated learning (FL) is a machine learning paradigm that targets model training without gathering the local data over various data sources. Standard FL, which employs a single server, can only support a limited number of users, leading to degraded learning capability. In this work, we consider a multi-server FL framework, referred to as emphConfederated Learning (CFL) in order to accommodate a larger number of users.
  arXiv  Detail & Related papers  (2024-02-28T03:27:10Z)
• Efficient Asynchronous Federated Learning with Sparsification and Quantization [55.6801207905772]
  Federated Learning (FL) is attracting more and more attention to collaboratively train a machine learning model without transferring raw data. FL generally exploits a parameter server and a large number of edge devices during the whole process of the model training. We propose TEASQ-Fed to exploit edge devices to asynchronously participate in the training process by actively applying for tasks.
  arXiv  Detail & Related papers  (2023-12-23T07:47:07Z)
• Adaptive Model Pruning and Personalization for Federated Learning over Wireless Networks [72.59891661768177]
  Federated learning (FL) enables distributed learning across edge devices while protecting data privacy. We consider a FL framework with partial model pruning and personalization to overcome these challenges. This framework splits the learning model into a global part with model pruning shared with all devices to learn data representations and a personalized part to be fine-tuned for a specific device.
  arXiv  Detail & Related papers  (2023-09-04T21:10:45Z)
• Scalable Collaborative Learning via Representation Sharing [53.047460465980144]
  Federated learning (FL) and Split Learning (SL) are two frameworks that enable collaborative learning while keeping the data private (on device) In FL, each data holder trains a model locally and releases it to a central server for aggregation. In SL, the clients must release individual cut-layer activations (smashed data) to the server and wait for its response (during both inference and back propagation). In this work, we present a novel approach for privacy-preserving machine learning, where the clients collaborate via online knowledge distillation using a contrastive loss.
  arXiv  Detail & Related papers  (2022-11-20T10:49:22Z)
• FedDM: Iterative Distribution Matching for Communication-Efficient Federated Learning [87.08902493524556]
  Federated learning(FL) has recently attracted increasing attention from academia and industry. We propose FedDM to build the global training objective from multiple local surrogate functions. In detail, we construct synthetic sets of data on each client to locally match the loss landscape from original data.
  arXiv  Detail & Related papers  (2022-07-20T04:55:18Z)
• Heterogeneous Ensemble Knowledge Transfer for Training Large Models in Federated Learning [22.310090483499035]
  Federated learning (FL) enables edge-devices to collaboratively learn a model without disclosing their private data to a central aggregating server. Most existing FL algorithms require models of identical architecture to be deployed across the clients and server. We propose a novel ensemble knowledge transfer method named Fed-ET in which small models are trained on clients, and used to train a larger model at the server.
  arXiv  Detail & Related papers  (2022-04-27T05:18:32Z)
• Federated Dynamic Sparse Training: Computing Less, Communicating Less, Yet Learning Better [88.28293442298015]
  Federated learning (FL) enables distribution of machine learning workloads from the cloud to resource-limited edge devices. We develop, implement, and experimentally validate a novel FL framework termed Federated Dynamic Sparse Training (FedDST) FedDST is a dynamic process that extracts and trains sparse sub-networks from the target full network.
  arXiv  Detail & Related papers  (2021-12-18T02:26:38Z)
• Comfetch: Federated Learning of Large Networks on Constrained Clients via Sketching [28.990067638230254]
  Federated learning (FL) is a popular paradigm for private and collaborative model training on the edge. We propose a novel algorithm, Comdirectional, which allows clients to train large networks using representations of the global neural network.
  arXiv  Detail & Related papers  (2021-09-17T04:48:42Z)
• Ternary Compression for Communication-Efficient Federated Learning [17.97683428517896]
  Federated learning provides a potential solution to privacy-preserving and secure machine learning. We propose a ternary federated averaging protocol (T-FedAvg) to reduce the upstream and downstream communication of federated learning systems. Our results show that the proposed T-FedAvg is effective in reducing communication costs and can even achieve slightly better performance on non-IID data.
  arXiv  Detail & Related papers  (2020-03-07T11:55:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.