Adaptive Hybrid Model Pruning in Federated Learning through Loss Exploration

• URL: http://arxiv.org/abs/2405.10271v2
• Date: Tue, 15 Oct 2024 12:06:07 GMT
• Title: Adaptive Hybrid Model Pruning in Federated Learning through Loss Exploration
• Authors: Christian Internò, Elena Raponi, Niki van Stein, Thomas Bäck, Markus Olhofer, Yaochu Jin, Barbara Hammer,
• Abstract summary: We introduce AutoFLIP, an innovative approach that utilizes a federated loss exploration phase to drive adaptive hybrid pruning. We show that AutoFLIP not only efficiently accelerates global convergence, but also achieves superior accuracy and robustness compared to traditional methods.
• Score: 17.589308358508863
• License:
• Abstract: The rapid proliferation of smart devices coupled with the advent of 6G networks has profoundly reshaped the domain of collaborative machine learning. Alongside growing privacy-security concerns in sensitive fields, these developments have positioned federated learning (FL) as a pivotal technology for decentralized model training. Despite its vast potential, specially in the age of complex foundation models, FL encounters challenges such as elevated communication costs, computational constraints, and the complexities of non-IID data distributions. We introduce AutoFLIP, an innovative approach that utilizes a federated loss exploration phase to drive adaptive hybrid pruning, operating in a structured and unstructured way. This innovative mechanism automatically identifies and prunes model substructure by distilling knowledge on model gradients behavior across different non-IID client losses topology, thereby optimizing computational efficiency and enhancing model performance on resource constrained scenarios. Extensive experiments on various datasets and FL tasks reveal that AutoFLIP not only efficiently accelerates global convergence, but also achieves superior accuracy and robustness compared to traditional methods. On average, AutoFLIP reduces computational overhead by 48.8% and communication costs by 35.5%, while improving global accuracy. By significantly reducing these overheads, AutoFLIP offer the way for efficient FL deployment in real-world applications for a scalable and broad applicability.

Related papers

• Online Client Scheduling and Resource Allocation for Efficient Federated Edge Learning [9.451084740123198]
  Federated learning (FL) enables edge devices to collaboratively train a machine learning model without sharing their raw data. However, deploying FL over mobile edge networks with constrained resources such as power, bandwidth, and suffers from high training latency and low model accuracy. This paper investigates the optimal client scheduling and resource allocation for FL over mobile edge networks under resource constraints and uncertainty.
  arXiv  Detail & Related papers  (2024-09-29T01:56:45Z)
• TriplePlay: Enhancing Federated Learning with CLIP for Non-IID Data and Resource Efficiency [0.0]
  TriplePlay is a framework that integrates CLIP as an adapter to enhance FL's adaptability and performance across diverse data distributions. Our simulation results demonstrate that TriplePlay effectively decreases GPU usage costs and speeds up the learning process, achieving convergence with reduced communication overhead.
  arXiv  Detail & Related papers  (2024-09-09T06:04:42Z)
• Hyperdimensional Computing Empowered Federated Foundation Model over Wireless Networks for Metaverse [56.384390765357004]
  We propose an integrated federated split learning and hyperdimensional computing framework for emerging foundation models. This novel approach reduces communication costs, computation load, and privacy risks, making it suitable for resource-constrained edge devices in the Metaverse.
  arXiv  Detail & Related papers  (2024-08-26T17:03:14Z)
• AEDFL: Efficient Asynchronous Decentralized Federated Learning with Heterogeneous Devices [61.66943750584406]
  We propose an Asynchronous Efficient Decentralized FL framework, i.e., AEDFL, in heterogeneous environments. First, we propose an asynchronous FL system model with an efficient model aggregation method for improving the FL convergence. Second, we propose a dynamic staleness-aware model update approach to achieve superior accuracy. Third, we propose an adaptive sparse training method to reduce communication and computation costs without significant accuracy degradation.
  arXiv  Detail & Related papers  (2023-12-18T05:18:17Z)
• Filling the Missing: Exploring Generative AI for Enhanced Federated Learning over Heterogeneous Mobile Edge Devices [72.61177465035031]
  We propose a generative AI-empowered federated learning to address these challenges by leveraging the idea of FIlling the MIssing (FIMI) portion of local data. Experiment results demonstrate that FIMI can save up to 50% of the device-side energy to achieve the target global test accuracy.
  arXiv  Detail & Related papers  (2023-10-21T12:07:04Z)
• Adaptive Model Pruning and Personalization for Federated Learning over Wireless Networks [72.59891661768177]
  Federated learning (FL) enables distributed learning across edge devices while protecting data privacy. We consider a FL framework with partial model pruning and personalization to overcome these challenges. This framework splits the learning model into a global part with model pruning shared with all devices to learn data representations and a personalized part to be fine-tuned for a specific device.
  arXiv  Detail & Related papers  (2023-09-04T21:10:45Z)
• Vertical Federated Learning over Cloud-RAN: Convergence Analysis and System Optimization [82.12796238714589]
  We propose a novel cloud radio access network (Cloud-RAN) based vertical FL system to enable fast and accurate model aggregation. We characterize the convergence behavior of the vertical FL algorithm considering both uplink and downlink transmissions. We establish a system optimization framework by joint transceiver and fronthaul quantization design, for which successive convex approximation and alternate convex search based system optimization algorithms are developed.
  arXiv  Detail & Related papers  (2023-05-04T09:26:03Z)
• HiFlash: Communication-Efficient Hierarchical Federated Learning with Adaptive Staleness Control and Heterogeneity-aware Client-Edge Association [38.99309610943313]
  Federated learning (FL) is a promising paradigm that enables collaboratively learning a shared model across massive clients. For many existing FL systems, clients need to frequently exchange model parameters of large data size with the remote cloud server directly via wide-area networks (WAN) We resort to the hierarchical federated learning paradigm of HiFL, which reaps the benefits of mobile edge computing.
  arXiv  Detail & Related papers  (2023-01-16T14:39:04Z)
• Communication-Efficient Diffusion Strategy for Performance Improvement of Federated Learning with Non-IID Data [10.994226932599403]
  We propose a novel diffusion strategy for machine learning (ML) models (FedDif) to maximize the performance of the global model with non-IID data. We show that FedDif improves the top-1 test accuracy by up to 34.89% and reduces communication costs by 14.6% to a maximum of 63.49%.
  arXiv  Detail & Related papers  (2022-07-15T14:28:41Z)
• Dynamic Attention-based Communication-Efficient Federated Learning [85.18941440826309]
  Federated learning (FL) offers a solution to train a global machine learning model. FL suffers performance degradation when client data distribution is non-IID. We propose a new adaptive training algorithm $textttAdaFL$ to combat this degradation.
  arXiv  Detail & Related papers  (2021-08-12T14:18:05Z)
• AutoFL: Enabling Heterogeneity-Aware Energy Efficient Federated Learning [7.802192899666384]
  Federated learning enables a cluster of decentralized mobile devices at the edge to collaboratively train a shared machine learning model. This decentralized training approach is demonstrated as a practical solution to mitigate the risk of privacy leakage. This paper jointly optimize time-to-convergence and energy efficiency of state-of-the-art FL use cases.
  arXiv  Detail & Related papers  (2021-07-16T23:41:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.