Communication-efficient Federated Learning with Single-Step Synthetic Features Compressor for Faster Convergence

• URL: http://arxiv.org/abs/2302.13562v2
• Date: Sun, 19 Mar 2023 03:45:40 GMT
• Title: Communication-efficient Federated Learning with Single-Step Synthetic Features Compressor for Faster Convergence
• Abstract summary: We propose single-step synthetic features compressor (3SFC) to achieve communication-efficient federated learning. 3SFC's compressing phase utilizes a similarity-based objective function so that it can be optimized with just one step. Experiments on multiple datasets and models suggest that 3SFC owns significantly better convergence rates compared to competing methods.
• Score: 18.424046602963926
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reducing communication overhead in federated learning (FL) is challenging but crucial for large-scale distributed privacy-preserving machine learning. While methods utilizing sparsification or others can largely lower the communication overhead, the convergence rate is also greatly compromised. In this paper, we propose a novel method, named single-step synthetic features compressor (3SFC), to achieve communication-efficient FL by directly constructing a tiny synthetic dataset based on raw gradients. Thus, 3SFC can achieve an extremely low compression rate when the constructed dataset contains only one data sample. Moreover, 3SFC's compressing phase utilizes a similarity-based objective function so that it can be optimized with just one step, thereby considerably improving its performance and robustness. In addition, to minimize the compressing error, error feedback (EF) is also incorporated into 3SFC. Experiments on multiple datasets and models suggest that 3SFC owns significantly better convergence rates compared to competing methods with lower compression rates (up to 0.02%). Furthermore, ablation studies and visualizations show that 3SFC can carry more information than competing methods for every communication round, further validating its effectiveness.

Related papers

• Harnessing Lightweight Transformer with Contextual Synergic Enhancement for Efficient 3D Medical Image Segmentation [59.711744386226194]
  We propose Light-UNETR, a lightweight transformer designed to achieve model efficiency.<n>Light-UNETR features a Lightweight Dimension Reductive Attention (LIDR) module, which reduces spatial and channel dimensions.<n>We also introduce a Contextual Synergic Enhancement (CSE) learning strategy, which aims to boost the data efficiency of Transformers.
  arXiv  Detail & Related papers  (2026-03-24T16:24:19Z)
• An Efficient Gradient-Aware Error-Bounded Lossy Compressor for Federated Learning [7.649286962189554]
  Federated learning (FL) enables collaborative model training without exposing clients' private data.<n>EBLC is particularly appealing for its fine-grained utility-compression tradeoff.<n>We propose an EBLC framework tailored for FL gradient data to achieve high compression ratios while preserving model accuracy.
  arXiv  Detail & Related papers  (2025-11-07T23:59:09Z)
• Gradient Projection onto Historical Descent Directions for Communication-Efficient Federated Learning [0.8220217498103312]
  Federated Learning (FL) enables decentralized model training across multiple clients while preserving data privacy.<n>We introduce two algorithms: ProjFL, designed for unbiased compressors, and ProjFL+EF, for biased compressors through an Error Feedback mechanism.
  arXiv  Detail & Related papers  (2025-11-05T13:11:30Z)
• High-Energy Concentration for Federated Learning in Frequency Domain [62.90597472173119]
  We propose a novel Frequency-Domain aware FL method with high-energy concentration (FedFD) to address this problem.<n>FedFD is inspired by the discovery that the discrete cosine transform predominantly distributes energy to specific regions, referred to as high-energy concentration.<n>On five image and speech datasets, FedFD achieves superior performance than state-of-the-art methods while reducing communication costs.
  arXiv  Detail & Related papers  (2025-09-16T03:49:26Z)
• E-3SFC: Communication-Efficient Federated Learning with Double-way Features Synthesizing [23.586844019776024]
  The exponential growth in model sizes has significantly increased the communication burden in Learning (FL) Existing methods to alleviate this burden by transmitting compressed gradients face high compression errors, which slow down the model's performance. We propose a systematical algorithm termed Extended Single-Step Synthetic Features. E-3SFC consists of three sub-components, i.e. the Single-Step Synthetic Features (3SFC), a double-way compression algorithm, and a communication budget scheduler.
  arXiv  Detail & Related papers  (2025-02-05T11:31:21Z)
• Foundation Model for Lossy Compression of Spatiotemporal Scientific Data [11.494915987840876]
  We present a foundation model (FM) for lossy scientific data compression. We combine a variational autoencoder (E) with a hyper-prior structure and a super-resolution (SR) module.
  arXiv  Detail & Related papers  (2024-12-22T22:57:08Z)
• TriplePlay: Enhancing Federated Learning with CLIP for Non-IID Data and Resource Efficiency [0.0]
  TriplePlay is a framework that integrates CLIP as an adapter to enhance FL's adaptability and performance across diverse data distributions. Our simulation results demonstrate that TriplePlay effectively decreases GPU usage costs and speeds up the learning process, achieving convergence with reduced communication overhead.
  arXiv  Detail & Related papers  (2024-09-09T06:04:42Z)
• Bandwidth-Aware and Overlap-Weighted Compression for Communication-Efficient Federated Learning [29.727339562140653]
  Current data compression methods, such as sparsification in Federated Averaging (FedAvg), effectively enhance the communication efficiency of Federated Learning (FL) These methods encounter challenges such as the straggler problem and diminished model performance due to heterogeneous bandwidth and non-IID data. We introduce a bandwidth-aware compression framework for FL, aimed at improving communication efficiency while mitigating the problems associated with non-IID data.
  arXiv  Detail & Related papers  (2024-08-27T02:28:27Z)
• FedComLoc: Communication-Efficient Distributed Training of Sparse and Quantized Models [56.21666819468249]
  Federated Learning (FL) has garnered increasing attention due to its unique characteristic of allowing heterogeneous clients to process their private data locally and interact with a central server. We introduce FedComLoc, integrating practical and effective compression into emphScaffnew to further enhance communication efficiency.
  arXiv  Detail & Related papers  (2024-03-14T22:29:59Z)
• Communication-Efficient Distributed Learning with Local Immediate Error Compensation [95.6828475028581]
  We propose the Local Immediate Error Compensated SGD (LIEC-SGD) optimization algorithm. LIEC-SGD is superior to previous works in either the convergence rate or the communication cost.
  arXiv  Detail & Related papers  (2024-02-19T05:59:09Z)
• Improving the Worst-Case Bidirectional Communication Complexity for Nonconvex Distributed Optimization under Function Similarity [92.1840862558718]
  We introduce MARINA-P, a novel method for downlink compression, employing a collection of correlated compressors. We show that MARINA-P with permutation compressors can achieve a server-to-worker communication complexity improving with the number of workers. We introduce M3, a method combining MARINA-P with uplink compression and a momentum step, achieving bidirectional compression with provable improvements in total communication complexity as the number of workers increases.
  arXiv  Detail & Related papers  (2024-02-09T13:58:33Z)
• Fed-CVLC: Compressing Federated Learning Communications with Variable-Length Codes [54.18186259484828]
  In Federated Learning (FL) paradigm, a parameter server (PS) concurrently communicates with distributed participating clients for model collection, update aggregation, and model distribution over multiple rounds. We show strong evidences that variable-length is beneficial for compression in FL. We present Fed-CVLC (Federated Learning Compression with Variable-Length Codes), which fine-tunes the code length in response to the dynamics of model updates.
  arXiv  Detail & Related papers  (2024-02-06T07:25:21Z)
• Stochastic Controlled Averaging for Federated Learning with Communication Compression [36.09135695005069]
  We propose two compressed FL algorithms, SCALLION and SCAFCOM, to support unbiased and biased compression. Both the proposed methods outperform the existing compressed FL methods in terms of communication and computation complexities.
  arXiv  Detail & Related papers  (2023-08-16T06:35:36Z)
• HFedMS: Heterogeneous Federated Learning with Memorable Data Semantics in Industrial Metaverse [49.1501082763252]
  This paper presents HFEDMS for incorporating practical FL into the emerging Industrial Metaverse. It reduces data heterogeneity through dynamic grouping and training mode conversion. Then, it compensates for the forgotten knowledge by fusing compressed historical data semantics. Experiments have been conducted on the streamed non-i.i.d. FEMNIST dataset using 368 simulated devices.
  arXiv  Detail & Related papers  (2022-11-07T04:33:24Z)
• CAFE: Learning to Condense Dataset by Aligning Features [72.99394941348757]
  We propose a novel scheme to Condense dataset by Aligning FEatures (CAFE) At the heart of our approach is an effective strategy to align features from the real and synthetic data across various scales. We validate the proposed CAFE across various datasets, and demonstrate that it generally outperforms the state of the art.
  arXiv  Detail & Related papers  (2022-03-03T05:58:49Z)
• Efficient Data Compression for 3D Sparse TPC via Bicephalous Convolutional Autoencoder [8.759778406741276]
  This work introduces a dual-head autoencoder to resolve sparsity and regression simultaneously, called textitBicephalous Convolutional AutoEncoder (BCAE) It shows advantages both in compression fidelity and ratio compared to traditional data compression methods, such as MGARD, SZ, and ZFP.
  arXiv  Detail & Related papers  (2021-11-09T21:26:37Z)
• Communication-Efficient Federated Learning with Compensated Overlap-FedAvg [22.636184975591004]
  Federated learning is proposed to perform model training by multiple clients' combined data without the dataset sharing within the cluster. We propose Overlap-FedAvg, a framework that parallels the model training phase with model uploading & downloading phase. Overlap-FedAvg is further developed with a hierarchical computing strategy, a data compensation mechanism and a nesterov accelerated gradients(NAG) algorithm.
  arXiv  Detail & Related papers  (2020-12-12T02:50:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.