Automatic Structured Pruning for Efficient Architecture in Federated Learning

• URL: http://arxiv.org/abs/2411.01759v1
• Date: Mon, 04 Nov 2024 02:52:02 GMT
• Title: Automatic Structured Pruning for Efficient Architecture in Federated Learning
• Authors: Thai Vu Nguyen, Long Bao Le, Anderson Avila,
• Abstract summary: In Federated Learning (FL), training is conducted on client devices, typically with limited computational resources and storage capacity. We propose an automatic pruning scheme tailored for FL systems. Our solution improves efficiency on client devices, while minimizing communication costs.
• Score: 5.300811350105823
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In Federated Learning (FL), training is conducted on client devices, typically with limited computational resources and storage capacity. To address these constraints, we propose an automatic pruning scheme tailored for FL systems. Our solution improves computation efficiency on client devices, while minimizing communication costs. One of the challenges of tuning pruning hyper-parameters in FL systems is the restricted access to local data. Thus, we introduce an automatic pruning paradigm that dynamically determines pruning boundaries. Additionally, we utilized a structured pruning algorithm optimized for mobile devices that lack hardware support for sparse computations. Experimental results demonstrate the effectiveness of our approach, achieving accuracy comparable to existing methods. Our method notably reduces the number of parameters by 89% and FLOPS by 90%, with minimal impact on the accuracy of the FEMNIST and CelebFaces datasets. Furthermore, our pruning method decreases communication overhead by up to 5x and halves inference time when deployed on Android devices.

Related papers

• Adaptive Rank Allocation for Federated Parameter-Efficient Fine-Tuning of Language Models [40.69348434971122]
  We propose FedARA, a novel Adaptive Rank Allocation framework for federated parameter-efficient fine-tuning of language models. FedARA consistently outperforms baselines by an average of 6.95% to 8.49% across various datasets and models under heterogeneous data. Experiments on various edge devices demonstrate substantial decreases in total training time and energy consumption by up to 48.90% and 46.95%, respectively.
  arXiv  Detail & Related papers  (2025-01-24T11:19:07Z)
• Efficient Deployment of Large Language Models on Resource-constrained Devices [12.644230479753476]
  It is necessary to fine-tune Large Language Models (LLMs) on resource-constrained devices for various downstream tasks. FedSpine is a framework that combines Efficient Fine-Tuning (PEFT) with structured pruning for efficient deployment of LLMs on resource-constrained devices. We show that FedSpine can speed up fine-tuning by 1.4times$$$times and improve final accuracy by 0.4%-4.5% under the same sparsity level compared to other baselines.
  arXiv  Detail & Related papers  (2025-01-05T04:38:11Z)
• A Web-Based Solution for Federated Learning with LLM-Based Automation [34.756818299081736]
  Federated Learning (FL) offers a promising approach for collaborative machine learning across distributed devices. We develop a user-friendly web application supporting the federated averaging (FedAvg) algorithm. We explore intent-based automation in FL using a fine-tuned Language Model (LLM) trained on a tailored dataset.
  arXiv  Detail & Related papers  (2024-08-23T11:57:02Z)
• Resource Management for Low-latency Cooperative Fine-tuning of Foundation Models at the Network Edge [35.40849522296486]
  Large-scale foundation models (FoMos) can perform human-like intelligence. FoMos need to be adapted to specialized downstream tasks through fine-tuning techniques. We advocate multi-device cooperation within the device-edge cooperative fine-tuning paradigm.
  arXiv  Detail & Related papers  (2024-07-13T12:47:14Z)
• SpaFL: Communication-Efficient Federated Learning with Sparse Models and Low computational Overhead [75.87007729801304]
  SpaFL: a communication-efficient FL framework is proposed to optimize sparse model structures with low computational overhead. Experiments show that SpaFL improves accuracy while requiring much less communication and computing resources compared to sparse baselines.
  arXiv  Detail & Related papers  (2024-06-01T13:10:35Z)
• FALCON: FLOP-Aware Combinatorial Optimization for Neural Network Pruning [17.60353530072587]
  Network pruning offers a solution to reduce model size and computational cost while maintaining performance. Most current pruning methods focus primarily on improving sparsity by reducing the number of nonzero parameters. We propose FALCON, a novel-optimization-based framework for network pruning that jointly takes into account model accuracy (fidelity), FLOPs, and sparsity constraints.
  arXiv  Detail & Related papers  (2024-03-11T18:40:47Z)
• Federated Learning of Large Language Models with Parameter-Efficient Prompt Tuning and Adaptive Optimization [71.87335804334616]
  Federated learning (FL) is a promising paradigm to enable collaborative model training with decentralized data. The training process of Large Language Models (LLMs) generally incurs the update of significant parameters. This paper proposes an efficient partial prompt tuning approach to improve performance and efficiency simultaneously.
  arXiv  Detail & Related papers  (2023-10-23T16:37:59Z)
• Adaptive Model Pruning and Personalization for Federated Learning over Wireless Networks [72.59891661768177]
  Federated learning (FL) enables distributed learning across edge devices while protecting data privacy. We consider a FL framework with partial model pruning and personalization to overcome these challenges. This framework splits the learning model into a global part with model pruning shared with all devices to learn data representations and a personalized part to be fine-tuned for a specific device.
  arXiv  Detail & Related papers  (2023-09-04T21:10:45Z)
• Gradient Sparsification for Efficient Wireless Federated Learning with Differential Privacy [25.763777765222358]
  Federated learning (FL) enables distributed clients to collaboratively train a machine learning model without sharing raw data with each other. As the model size grows, the training latency due to limited transmission bandwidth and private information degrades while using differential privacy (DP) protection. We propose sparsification empowered FL framework wireless channels, in over to improve training efficiency without sacrificing convergence performance.
  arXiv  Detail & Related papers  (2023-04-09T05:21:15Z)
• AnycostFL: Efficient On-Demand Federated Learning over Heterogeneous Edge Devices [20.52519915112099]
  We propose a cost-adjustable FL framework, named AnycostFL, that enables diverse edge devices to efficiently perform local updates. Experiment results indicate that, our learning framework can reduce up to 1.9 times of the training latency and energy consumption for realizing a reasonable global testing accuracy.
  arXiv  Detail & Related papers  (2023-01-08T15:25:55Z)
• Efficient Micro-Structured Weight Unification and Pruning for Neural Network Compression [56.83861738731913]
  Deep Neural Network (DNN) models are essential for practical applications, especially for resource limited devices. Previous unstructured or structured weight pruning methods can hardly truly accelerate inference. We propose a generalized weight unification framework at a hardware compatible micro-structured level to achieve high amount of compression and acceleration.
  arXiv  Detail & Related papers  (2021-06-15T17:22:59Z)
• Edge Federated Learning Via Unit-Modulus Over-The-Air Computation (Extended Version) [64.76619508293966]
  This paper proposes a unit-modulus over-the-air computation (UM-AirComp) framework to facilitate efficient edge federated learning. It uploads simultaneously local model parameters and updates global model parameters via analog beamforming. We demonstrate the implementation of UM-AirComp in a vehicle-to-everything autonomous driving simulation platform.
  arXiv  Detail & Related papers  (2021-01-28T15:10:22Z)
• ALF: Autoencoder-based Low-rank Filter-sharing for Efficient Convolutional Neural Networks [63.91384986073851]
  We propose the autoencoder-based low-rank filter-sharing technique technique (ALF) ALF shows a reduction of 70% in network parameters, 61% in operations and 41% in execution time, with minimal loss in accuracy.
  arXiv  Detail & Related papers  (2020-07-27T09:01:22Z)
• A Privacy-Preserving-Oriented DNN Pruning and Mobile Acceleration Framework [56.57225686288006]
  Weight pruning of deep neural networks (DNNs) has been proposed to satisfy the limited storage and computing capability of mobile edge devices. Previous pruning methods mainly focus on reducing the model size and/or improving performance without considering the privacy of user data. We propose a privacy-preserving-oriented pruning and mobile acceleration framework that does not require the private training dataset.
  arXiv  Detail & Related papers  (2020-03-13T23:52:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.