Adaptive Differential Filters for Fast and Communication-Efficient Federated Learning

• URL: http://arxiv.org/abs/2204.04424v1
• Date: Sat, 9 Apr 2022 08:23:25 GMT
• Title: Adaptive Differential Filters for Fast and Communication-Efficient Federated Learning
• Authors: Daniel Becking and Heiner Kirchhoffer and Gerhard Tech and Paul Haase and Karsten M\"uller and Heiko Schwarz and Wojciech Samek
• Abstract summary: Federated learning (FL) scenarios generate a large communication overhead by frequently transmitting neural network updates between clients and server. We propose a new scaling method operating at the granularity of convolutional filters which compensates for sparse updates in FL processes. The proposed method improves the performance of the central server model while converging faster and reducing the total amount of transmitted data by up to 377 times.
• Score: 12.067586493399308
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Federated learning (FL) scenarios inherently generate a large communication overhead by frequently transmitting neural network updates between clients and server. To minimize the communication cost, introducing sparsity in conjunction with differential updates is a commonly used technique. However, sparse model updates can slow down convergence speed or unintentionally skip certain update aspects, e.g., learned features, if error accumulation is not properly addressed. In this work, we propose a new scaling method operating at the granularity of convolutional filters which 1) compensates for highly sparse updates in FL processes, 2) adapts the local models to new data domains by enhancing some features in the filter space while diminishing others and 3) motivates extra sparsity in updates and thus achieves higher compression ratios, i.e., savings in the overall data transfer. Compared to unscaled updates and previous work, experimental results on different computer vision tasks (Pascal VOC, CIFAR10, Chest X-Ray) and neural networks (ResNets, MobileNets, VGGs) in uni-, bidirectional and partial update FL settings show that the proposed method improves the performance of the central server model while converging faster and reducing the total amount of transmitted data by up to 377 times.

Related papers

• Communication-Efficient Federated Learning Based on Explanation-Guided Pruning for Remote Sensing Image Classification [2.725507329935916]
  We introduce an explanation-guided pruning strategy for communication-efficient Federated Learning (FL) Our strategy effectively reduces the number of shared model updates, while increasing the ability of the global model. The code of this work will be publicly available at https://git.tu-berlin.de/rsim/FL-LRP.
  arXiv  Detail & Related papers  (2025-01-20T13:59:41Z)
• Reducing Data Bottlenecks in Distributed, Heterogeneous Neural Networks [5.32129361961937]
  This paper investigates the impact of bottleneck size on the performance of deep learning models in embedded multicore and many-core systems. We apply a hardware-software co-design methodology where data bottlenecks are replaced with extremely narrow layers to reduce the amount of data traffic. Hardware-side evaluation reveals that higher bottleneck ratios lead to substantial reductions in data transfer volume across the layers of the neural network.
  arXiv  Detail & Related papers  (2024-10-12T21:07:55Z)
• Efficient Federated Learning Using Dynamic Update and Adaptive Pruning with Momentum on Shared Server Data [59.6985168241067]
  Federated Learning (FL) encounters two important problems, i.e., low training efficiency and limited computational resources. We propose a new FL framework, FedDUMAP, to leverage the shared insensitive data on the server and the distributed data in edge devices. Our proposed FL model, FedDUMAP, combines the three original techniques and has a significantly better performance compared with baseline approaches.
  arXiv  Detail & Related papers  (2024-08-11T02:59:11Z)
• High-Dimensional Distributed Sparse Classification with Scalable Communication-Efficient Global Updates [50.406127962933915]
  We develop solutions to problems which enable us to learn a communication-efficient distributed logistic regression model. In our experiments we demonstrate a large improvement in accuracy over distributed algorithms with only a few distributed update steps needed.
  arXiv  Detail & Related papers  (2024-07-08T19:34:39Z)
• Over-the-Air Federated Learning and Optimization [52.5188988624998]
  We focus on Federated learning (FL) via edge-the-air computation (AirComp) We describe the convergence of AirComp-based FedAvg (AirFedAvg) algorithms under both convex and non- convex settings. For different types of local updates that can be transmitted by edge devices (i.e., model, gradient, model difference), we reveal that transmitting in AirFedAvg may cause an aggregation error. In addition, we consider more practical signal processing schemes to improve the communication efficiency and extend the convergence analysis to different forms of model aggregation error caused by these signal processing schemes.
  arXiv  Detail & Related papers  (2023-10-16T05:49:28Z)
• FLARE: Detection and Mitigation of Concept Drift for Federated Learning based IoT Deployments [2.7776688429637466]
  FLARE is a lightweight dual-scheduler FL framework that conditionally transfers training data and deploys models between edge and sensor endpoints. We show that FLARE can significantly reduce the amount of data exchanged between edge and sensor nodes compared to fixed-interval scheduling methods. It can successfully detect concept drift reactively with at least a 16x reduction in latency.
  arXiv  Detail & Related papers  (2023-05-15T10:09:07Z)
• Federated Progressive Sparsification (Purge, Merge, Tune)+ [15.08232397899507]
  FedSparsify is a sparsification strategy based on progressive weight magnitude pruning. We show experimentally that FedSparsify learns a subnetwork of both high sparsity and learning performance.
  arXiv  Detail & Related papers  (2022-04-26T16:45:53Z)
• Acceleration of Federated Learning with Alleviated Forgetting in Local Training [61.231021417674235]
  Federated learning (FL) enables distributed optimization of machine learning models while protecting privacy. We propose FedReg, an algorithm to accelerate FL with alleviated knowledge forgetting in the local training stage. Our experiments demonstrate that FedReg not only significantly improves the convergence rate of FL, especially when the neural network architecture is deep.
  arXiv  Detail & Related papers  (2022-03-05T02:31:32Z)
• Federated Dynamic Sparse Training: Computing Less, Communicating Less, Yet Learning Better [88.28293442298015]
  Federated learning (FL) enables distribution of machine learning workloads from the cloud to resource-limited edge devices. We develop, implement, and experimentally validate a novel FL framework termed Federated Dynamic Sparse Training (FedDST) FedDST is a dynamic process that extracts and trains sparse sub-networks from the target full network.
  arXiv  Detail & Related papers  (2021-12-18T02:26:38Z)
• Slashing Communication Traffic in Federated Learning by Transmitting Clustered Model Updates [12.660500431713336]
  Federated Learning (FL) is an emerging decentralized learning framework through which multiple clients can collaboratively train a learning model. heavy communication traffic can be incurred by exchanging model updates via the Internet between clients and the parameter server. In this work, we devise the Model Update Compression by Soft Clustering (MUCSC) algorithm to compress model updates transmitted between clients and the PS.
  arXiv  Detail & Related papers  (2021-05-10T07:15:49Z)
• FedAT: A High-Performance and Communication-Efficient Federated Learning System with Asynchronous Tiers [22.59875034596411]
  We present FedAT, a novel Federated learning method with Asynchronous Tiers under Non-i.i.d. data. FedAT minimizes the straggler effect with improved convergence speed and test accuracy. Results show that FedAT improves the prediction performance by up to 21.09%, and reduces the communication cost by up to 8.5x, compared to state-of-the-art FL methods.
  arXiv  Detail & Related papers  (2020-10-12T18:38:51Z)
• Over-the-Air Federated Learning from Heterogeneous Data [107.05618009955094]
  Federated learning (FL) is a framework for distributed learning of centralized models. We develop a Convergent OTA FL (COTAF) algorithm which enhances the common local gradient descent (SGD) FL algorithm. We numerically show that the precoding induced by COTAF notably improves the convergence rate and the accuracy of models trained via OTA FL.
  arXiv  Detail & Related papers  (2020-09-27T08:28:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.