FedPaI: Achieving Extreme Sparsity in Federated Learning via Pruning at Initialization

• URL: http://arxiv.org/abs/2504.00308v1
• Date: Tue, 01 Apr 2025 00:24:34 GMT
• Title: FedPaI: Achieving Extreme Sparsity in Federated Learning via Pruning at Initialization
• Authors: Haonan Wang, Zeli Liu, Kajimusugura Hoshino, Tuo Zhang, John Paul Walters, Stephen Crago,
• Abstract summary: Federated Learning (FL) enables distributed training on edge devices.<n>Current iterative pruning techniques improve communication efficiency but are limited by their centralized design.<n>We propose FedPaI, a novel efficient FL framework that leverages Pruning at Initialization (PaI) to achieve extreme sparsity.
• Score: 10.425903190996785
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Federated Learning (FL) enables distributed training on edge devices but faces significant challenges due to resource constraints in edge environments, impacting both communication and computational efficiency. Existing iterative pruning techniques improve communication efficiency but are limited by their centralized design, which struggles with FL's decentralized and data-imbalanced nature, resulting in suboptimal sparsity levels. To address these issues, we propose FedPaI, a novel efficient FL framework that leverages Pruning at Initialization (PaI) to achieve extreme sparsity. FedPaI identifies optimal sparse connections at an early stage, maximizing model capacity and significantly reducing communication and computation overhead by fixing sparsity patterns at the start of training. To adapt to diverse hardware and software environments, FedPaI supports both structured and unstructured pruning. Additionally, we introduce personalized client-side pruning mechanisms for improved learning capacity and sparsity-aware server-side aggregation for enhanced efficiency. Experimental results demonstrate that FedPaI consistently outperforms existing efficient FL that applies conventional iterative pruning with significant leading in efficiency and model accuracy. For the first time, our proposed FedPaI achieves an extreme sparsity level of up to 98% without compromising the model accuracy compared to unpruned baselines, even under challenging non-IID settings. By employing our FedPaI with joint optimization of model learning capacity and sparsity, FL applications can benefit from faster convergence and accelerate the training by 6.4 to 7.9 times.

Related papers

• Meta-Computing Enhanced Federated Learning in IIoT: Satisfaction-Aware Incentive Scheme via DRL-Based Stackelberg Game [50.6166553799783]
  Efficient IIoT operations require a trade-off between model quality and training latency. This paper designs a satisfaction function that accounts for data size, Age of Information (AoI), and training latency for meta-computing. We employ a deep reinforcement learning approach to learn the Stackelberg equilibrium.
  arXiv  Detail & Related papers  (2025-02-10T03:33:36Z)
• Heterogeneity-Aware Resource Allocation and Topology Design for Hierarchical Federated Edge Learning [9.900317349372383]
  Federated Learning (FL) provides a privacy-preserving framework for training machine learning models on mobile edge devices. Traditional FL algorithms, e.g., FedAvg, impose a heavy communication workload on these devices. We propose a two-tier HFEL system, where edge devices are connected to edge servers and edge servers are interconnected through peer-to-peer (P2P) edge backhauls. Our goal is to enhance the training efficiency of the HFEL system through strategic resource allocation and topology design.
  arXiv  Detail & Related papers  (2024-09-29T01:48:04Z)
• UniPTS: A Unified Framework for Proficient Post-Training Sparsity [67.16547529992928]
  Post-training Sparsity (PTS) is a newly emerged avenue that chases efficient network sparsity with limited data in need. In this paper, we attempt to reconcile this disparity by transposing three cardinal factors that profoundly alter the performance of conventional sparsity into the context of PTS. Our framework, termed UniPTS, is validated to be much superior to existing PTS methods across extensive benchmarks.
  arXiv  Detail & Related papers  (2024-05-29T06:53:18Z)
• AEDFL: Efficient Asynchronous Decentralized Federated Learning with Heterogeneous Devices [61.66943750584406]
  We propose an Asynchronous Efficient Decentralized FL framework, i.e., AEDFL, in heterogeneous environments. First, we propose an asynchronous FL system model with an efficient model aggregation method for improving the FL convergence. Second, we propose a dynamic staleness-aware model update approach to achieve superior accuracy. Third, we propose an adaptive sparse training method to reduce communication and computation costs without significant accuracy degradation.
  arXiv  Detail & Related papers  (2023-12-18T05:18:17Z)
• Federated Learning of Large Language Models with Parameter-Efficient Prompt Tuning and Adaptive Optimization [71.87335804334616]
  Federated learning (FL) is a promising paradigm to enable collaborative model training with decentralized data. The training process of Large Language Models (LLMs) generally incurs the update of significant parameters. This paper proposes an efficient partial prompt tuning approach to improve performance and efficiency simultaneously.
  arXiv  Detail & Related papers  (2023-10-23T16:37:59Z)
• FLrce: Resource-Efficient Federated Learning with Early-Stopping Strategy [7.963276533979389]
  Federated Learning (FL) achieves great popularity in the Internet of Things (IoT) We present FLrce, an efficient FL framework with a relationship-based client selection and early-stopping strategy. Experiment results show that, compared with existing efficient FL frameworks, FLrce improves the computation and communication efficiency by at least 30% and 43% respectively.
  arXiv  Detail & Related papers  (2023-10-15T10:13:44Z)
• AnycostFL: Efficient On-Demand Federated Learning over Heterogeneous Edge Devices [20.52519915112099]
  We propose a cost-adjustable FL framework, named AnycostFL, that enables diverse edge devices to efficiently perform local updates. Experiment results indicate that, our learning framework can reduce up to 1.9 times of the training latency and energy consumption for realizing a reasonable global testing accuracy.
  arXiv  Detail & Related papers  (2023-01-08T15:25:55Z)
• FedDUAP: Federated Learning with Dynamic Update and Adaptive Pruning Using Shared Data on the Server [64.94942635929284]
  Federated Learning (FL) suffers from two critical challenges, i.e., limited computational resources and low training efficiency. We propose a novel FL framework, FedDUAP, to exploit the insensitive data on the server and the decentralized data in edge devices. By integrating the two original techniques together, our proposed FL model, FedDUAP, significantly outperforms baseline approaches in terms of accuracy (up to 4.8% higher), efficiency (up to 2.8 times faster), and computational cost (up to 61.9% smaller)
  arXiv  Detail & Related papers  (2022-04-25T10:00:00Z)
• Federated Dynamic Sparse Training: Computing Less, Communicating Less, Yet Learning Better [88.28293442298015]
  Federated learning (FL) enables distribution of machine learning workloads from the cloud to resource-limited edge devices. We develop, implement, and experimentally validate a novel FL framework termed Federated Dynamic Sparse Training (FedDST) FedDST is a dynamic process that extracts and trains sparse sub-networks from the target full network.
  arXiv  Detail & Related papers  (2021-12-18T02:26:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.