Breaking the Memory Wall for Heterogeneous Federated Learning via Model Splitting

• URL: http://arxiv.org/abs/2410.11577v1
• Date: Sat, 12 Oct 2024 18:23:21 GMT
• Title: Breaking the Memory Wall for Heterogeneous Federated Learning via Model Splitting
• Authors: Chunlin Tian, Li Li, Kahou Tam, Yebo Wu, Chengzhong Xu,
• Abstract summary: Federated Learning (FL) enables multiple devices to collaboratively train a shared model while preserving data privacy. Ever-increasing model complexity coupled with limited memory resources on the participating devices severely bottlenecks the deployment of FL in real-world scenarios. We propose SmartSplit, a framework that effectively reduces the memory footprint on the device side while guaranteeing the training progress and model accuracy for heterogeneous FL.
• Score: 16.42580791094151
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Federated Learning (FL) enables multiple devices to collaboratively train a shared model while preserving data privacy. Ever-increasing model complexity coupled with limited memory resources on the participating devices severely bottlenecks the deployment of FL in real-world scenarios. Thus, a framework that can effectively break the memory wall while jointly taking into account the hardware and statistical heterogeneity in FL is urgently required. In this paper, we propose SmartSplit, a framework that effectively reduces the memory footprint on the device side while guaranteeing the training progress and model accuracy for heterogeneous FL through model splitting.Towards this end, SmartSplit employs a hierarchical structure to adaptively guide the overall training process. In each training round, the central manager, hosted on the server, dynamically selects the participating devices and sets the cutting layer by jointly considering the memory budget, training capacity, and data distribution of each device. The MEC manager, deployed within the edge server, proceeds to split the local model and perform training of the server-side portion. Meanwhile, it fine-tunes the splitting points based on the time-evolving statistical importance. The on-device manager, embedded inside each mobile device, continuously monitors the local training status while employing cost-aware checkpointing to match the runtime dynamic memory budget. Extensive experiments on representative datasets are conducted on both commercial off-the-shelf mobile device testbeds. The experimental results show that SmartSplit excels in FL training on highly memory-constrained mobile SoCs, offering up to a 94% peak latency reduction and 100-fold memory savings. It enhances accuracy performance by 1.49%-57.18% and adaptively adjusts to dynamic memory budgets through cost-aware recomputation.

Related papers

• FedHybrid: Breaking the Memory Wall of Federated Learning via Hybrid Tensor Management [27.731967925365954]
  Federated Learning (FL) emerges as a new learning paradigm that enables multiple devices to collaboratively train a shared model.<n>One fundamental and prevailing challenge that hinders the deployment of FL on mobile devices is the memory limitation.<n>This paper proposes textitFedHybrid, a novel framework that effectively reduces the memory footprint during the training process.
  arXiv  Detail & Related papers  (2025-10-13T13:43:55Z)
• CollaPipe: Adaptive Segment-Optimized Pipeline Parallelism for Collaborative LLM Training in Heterogeneous Edge Networks [57.95170323315603]
  We introduce CollaPipe, a distributed learning framework that integrates collaborative pipeline parallelism with federated aggregation to support self-evolving networks.<n>In CollaPipe, the encoder part is adaptively partitioned into variable-sized segments and deployed across mobile devices for pipeline-parallel training, while the decoder is deployed on edge servers to handle generative tasks.<n>To enhance training efficiency, we formulate a joint optimization problem that adaptively allocates model segments, micro-batches, bandwidth, and transmission power.
  arXiv  Detail & Related papers  (2025-09-24T07:54:01Z)
• CaBaFL: Asynchronous Federated Learning via Hierarchical Cache and Feature Balance [23.125185494897522]
  Federated Learning (FL) as a promising distributed machine learning paradigm has been widely adopted in Artificial Intelligence of Things (AIoT) applications. The efficiency and inference capability of FL is seriously limited due to the presence of stragglers and data imbalance across massive AIoT devices. We present a novel FL approach named CaBaFL, which includes a hierarchical Cache-based aggregation mechanism and a feature Balance-guided device selection strategy.
  arXiv  Detail & Related papers  (2024-04-19T12:39:11Z)
• Efficient Asynchronous Federated Learning with Sparsification and Quantization [55.6801207905772]
  Federated Learning (FL) is attracting more and more attention to collaboratively train a machine learning model without transferring raw data. FL generally exploits a parameter server and a large number of edge devices during the whole process of the model training. We propose TEASQ-Fed to exploit edge devices to asynchronously participate in the training process by actively applying for tasks.
  arXiv  Detail & Related papers  (2023-12-23T07:47:07Z)
• Adaptive Model Pruning and Personalization for Federated Learning over Wireless Networks [72.59891661768177]
  Federated learning (FL) enables distributed learning across edge devices while protecting data privacy. We consider a FL framework with partial model pruning and personalization to overcome these challenges. This framework splits the learning model into a global part with model pruning shared with all devices to learn data representations and a personalized part to be fine-tuned for a specific device.
  arXiv  Detail & Related papers  (2023-09-04T21:10:45Z)
• Memory-adaptive Depth-wise Heterogenous Federated Learning [24.13198329419849]
  We introduce a memory-adaptive depth-wise learning solution in FL called FeDepth, which adaptively decomposes the full model into blocks according to the memory budgets of each client. Our method outperforms state-of-the-art approaches, achieving 5% and more than 10% improvements in top-1 accuracy on CIFAR-10 and CIFAR-100, respectively.
  arXiv  Detail & Related papers  (2023-03-08T20:52:57Z)
• FedDCT: Federated Learning of Large Convolutional Neural Networks on Resource Constrained Devices using Divide and Collaborative Training [13.072061144206097]
  We introduce FedDCT, a novel distributed learning paradigm that enables the usage of large, high-performance CNNs on resource-limited edge devices. We empirically conduct experiments on standardized datasets, including CIFAR-10, CIFAR-100, and two real-world medical datasets HAM10000 and VAIPE. Compared to other existing approaches, FedDCT achieves higher accuracy and substantially reduces the number of communication rounds.
  arXiv  Detail & Related papers  (2022-11-20T11:11:56Z)
• Scalable Collaborative Learning via Representation Sharing [53.047460465980144]
  Federated learning (FL) and Split Learning (SL) are two frameworks that enable collaborative learning while keeping the data private (on device) In FL, each data holder trains a model locally and releases it to a central server for aggregation. In SL, the clients must release individual cut-layer activations (smashed data) to the server and wait for its response (during both inference and back propagation). In this work, we present a novel approach for privacy-preserving machine learning, where the clients collaborate via online knowledge distillation using a contrastive loss.
  arXiv  Detail & Related papers  (2022-11-20T10:49:22Z)
• Online Data Selection for Federated Learning with Limited Storage [53.46789303416799]
  Federated Learning (FL) has been proposed to achieve distributed machine learning among networked devices. The impact of on-device storage on the performance of FL is still not explored. In this work, we take the first step to consider the online data selection for FL with limited on-device storage.
  arXiv  Detail & Related papers  (2022-09-01T03:27:33Z)
• Multi-Edge Server-Assisted Dynamic Federated Learning with an Optimized Floating Aggregation Point [51.47520726446029]
  cooperative edge learning (CE-FL) is a distributed machine learning architecture. We model the processes taken during CE-FL, and conduct analytical training. We show the effectiveness of our framework with the data collected from a real-world testbed.
  arXiv  Detail & Related papers  (2022-03-26T00:41:57Z)
• Federated Dynamic Sparse Training: Computing Less, Communicating Less, Yet Learning Better [88.28293442298015]
  Federated learning (FL) enables distribution of machine learning workloads from the cloud to resource-limited edge devices. We develop, implement, and experimentally validate a novel FL framework termed Federated Dynamic Sparse Training (FedDST) FedDST is a dynamic process that extracts and trains sparse sub-networks from the target full network.
  arXiv  Detail & Related papers  (2021-12-18T02:26:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.