E-3SFC: Communication-Efficient Federated Learning with Double-way Features Synthesizing

• URL: http://arxiv.org/abs/2502.03092v1
• Date: Wed, 05 Feb 2025 11:31:21 GMT
• Title: E-3SFC: Communication-Efficient Federated Learning with Double-way Features Synthesizing
• Abstract summary: The exponential growth in model sizes has significantly increased the communication burden in Learning (FL)<n>Existing methods to alleviate this burden by transmitting compressed gradients face high compression errors, which slow down the model's performance.<n>We propose a systematical algorithm termed Extended Single-Step Synthetic Features.<n>E-3SFC consists of three sub-components, i.e. the Single-Step Synthetic Features (3SFC), a double-way compression algorithm, and a communication budget scheduler.
• Score: 23.586844019776024
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: The exponential growth in model sizes has significantly increased the communication burden in Federated Learning (FL). Existing methods to alleviate this burden by transmitting compressed gradients often face high compression errors, which slow down the model's convergence. To simultaneously achieve high compression effectiveness and lower compression errors, we study the gradient compression problem from a novel perspective. Specifically, we propose a systematical algorithm termed Extended Single-Step Synthetic Features Compressing (E-3SFC), which consists of three sub-components, i.e., the Single-Step Synthetic Features Compressor (3SFC), a double-way compression algorithm, and a communication budget scheduler. First, we regard the process of gradient computation of a model as decompressing gradients from corresponding inputs, while the inverse process is considered as compressing the gradients. Based on this, we introduce a novel gradient compression method termed 3SFC, which utilizes the model itself as a decompressor, leveraging training priors such as model weights and objective functions. 3SFC compresses raw gradients into tiny synthetic features in a single-step simulation, incorporating error feedback to minimize overall compression errors. To further reduce communication overhead, 3SFC is extended to E-3SFC, allowing double-way compression and dynamic communication budget scheduling. Our theoretical analysis under both strongly convex and non-convex conditions demonstrates that 3SFC achieves linear and sub-linear convergence rates with aggregation noise. Extensive experiments across six datasets and six models reveal that 3SFC outperforms state-of-the-art methods by up to 13.4% while reducing communication costs by 111.6 times. These findings suggest that 3SFC can significantly enhance communication efficiency in FL without compromising model performance.

Related papers

• Arbitrary Ratio Feature Compression via Next Token Prediction [52.10426317889982]
  Arbitrary Ratio Feature Compression (ARFC) framework supports any compression ratio with a single model.<n>ARC is an auto-regressive model that performs compression via next-gressive prediction.<n>MoS module refines the compressed tokens by utilizing multiple compression results.<n>ERGC is integrated into the training process to preserve semantic and structural relationships during compression.
  arXiv  Detail & Related papers  (2026-02-12T02:38:57Z)
• CSGaussian: Progressive Rate-Distortion Compression and Segmentation for 3D Gaussian Splatting [57.73006852239138]
  We present the first unified framework for rate-distortion-optimized compression and segmentation of 3D Gaussian Splatting (3DGS)<n>Inspired by recent advances in rate-distortion-optimized 3DGS compression, this work integrates semantic learning into the compression pipeline to support decoder-side applications.<n>Our scheme features a lightweight implicit neural representation-based hyperprior, enabling efficient entropy coding of both color and semantic attributes.
  arXiv  Detail & Related papers  (2026-01-19T08:21:45Z)
• FLaTEC: Frequency-Disentangled Latent Triplanes for Efficient Compression of LiDAR Point Clouds [52.997038111673966]
  FLaTEC is a frequency-aware compression model that enables the compression of a full scan with high compression ratios.<n>We convert voxelized embeddings into triplane representations to reduce sparsity, computational cost, and storage requirements.<n>Our method achieves state-of-the-art rate-distortion performance and outperforms the standard codecs by 78% and 94% in BD-rate on both datasets.
  arXiv  Detail & Related papers  (2025-11-25T08:37:49Z)
• A Systematic Study of Compression Ordering for Large Language Models [0.5926203312586109]
  This study systematically examines how knowledge distillation, structured pruning, and low-bit quantization perform when applied to the Qwen2.5 3B model.<n>Experiments show that quantization provides the greatest standalone compression, while pruning introduces moderate quality degradation.
  arXiv  Detail & Related papers  (2025-11-23T12:46:56Z)
• EDGC: Entropy-driven Dynamic Gradient Compression for Efficient LLM Training [11.898292264147427]
  Training large language models (LLMs) poses significant challenges regarding computational resources and memory capacity.<n>Existing approaches primarily rely on static gradient compression to enhance communication efficiency.<n>We propose an entropy-driven dynamic gradient compression framework called EDGC.
  arXiv  Detail & Related papers  (2025-11-13T14:05:34Z)
• An Efficient Gradient-Aware Error-Bounded Lossy Compressor for Federated Learning [7.649286962189554]
  Federated learning (FL) enables collaborative model training without exposing clients' private data.<n>EBLC is particularly appealing for its fine-grained utility-compression tradeoff.<n>We propose an EBLC framework tailored for FL gradient data to achieve high compression ratios while preserving model accuracy.
  arXiv  Detail & Related papers  (2025-11-07T23:59:09Z)
• Breaking the Compression Ceiling: Data-Free Pipeline for Ultra-Efficient Delta Compression [53.08742231761896]
  UltraDelta is a data-free delta compression pipeline that achieves both ultra-high compression and strong performance.<n>UltraDelta is designed to minimize redundancy, maximize information, and stabilize performance across inter-layer, intra-layer, and global dimensions.
  arXiv  Detail & Related papers  (2025-05-19T10:37:22Z)
• Foundation Model for Lossy Compression of Spatiotemporal Scientific Data [11.494915987840876]
  We present a foundation model (FM) for lossy scientific data compression. We combine a variational autoencoder (E) with a hyper-prior structure and a super-resolution (SR) module.
  arXiv  Detail & Related papers  (2024-12-22T22:57:08Z)
• Fast Feedforward 3D Gaussian Splatting Compression [55.149325473447384]
  3D Gaussian Splatting (FCGS) is an optimization-free model that can compress 3DGS representations rapidly in a single feed-forward pass. FCGS achieves a compression ratio of over 20X while maintaining fidelity, surpassing most per-scene SOTA optimization-based methods.
  arXiv  Detail & Related papers  (2024-10-10T15:13:08Z)
• Bandwidth-Aware and Overlap-Weighted Compression for Communication-Efficient Federated Learning [29.727339562140653]
  Current data compression methods, such as sparsification in Federated Averaging (FedAvg), effectively enhance the communication efficiency of Federated Learning (FL) These methods encounter challenges such as the straggler problem and diminished model performance due to heterogeneous bandwidth and non-IID data. We introduce a bandwidth-aware compression framework for FL, aimed at improving communication efficiency while mitigating the problems associated with non-IID data.
  arXiv  Detail & Related papers  (2024-08-27T02:28:27Z)
• Communication-Efficient Distributed Learning with Local Immediate Error Compensation [95.6828475028581]
  We propose the Local Immediate Error Compensated SGD (LIEC-SGD) optimization algorithm. LIEC-SGD is superior to previous works in either the convergence rate or the communication cost.
  arXiv  Detail & Related papers  (2024-02-19T05:59:09Z)
• Improving the Worst-Case Bidirectional Communication Complexity for Nonconvex Distributed Optimization under Function Similarity [92.1840862558718]
  We introduce MARINA-P, a novel method for downlink compression, employing a collection of correlated compressors. We show that MARINA-P with permutation compressors can achieve a server-to-worker communication complexity improving with the number of workers. We introduce M3, a method combining MARINA-P with uplink compression and a momentum step, achieving bidirectional compression with provable improvements in total communication complexity as the number of workers increases.
  arXiv  Detail & Related papers  (2024-02-09T13:58:33Z)
• Stochastic Controlled Averaging for Federated Learning with Communication Compression [36.09135695005069]
  We propose two compressed FL algorithms, SCALLION and SCAFCOM, to support unbiased and biased compression. Both the proposed methods outperform the existing compressed FL methods in terms of communication and computation complexities.
  arXiv  Detail & Related papers  (2023-08-16T06:35:36Z)
• Communication-efficient Federated Learning with Single-Step Synthetic Features Compressor for Faster Convergence [18.424046602963926]
  We propose single-step synthetic features compressor (3SFC) to achieve communication-efficient federated learning. 3SFC's compressing phase utilizes a similarity-based objective function so that it can be optimized with just one step. Experiments on multiple datasets and models suggest that 3SFC owns significantly better convergence rates compared to competing methods.
  arXiv  Detail & Related papers  (2023-02-27T07:47:23Z)
• Optimal Rate Adaption in Federated Learning with Compressed Communications [28.16239232265479]
  Federated Learning incurs high communication overhead, which can be greatly alleviated by compression for model updates. tradeoff between compression and model accuracy in the networked environment remains unclear. We present a framework to maximize the final model accuracy by strategically adjusting the compression each iteration.
  arXiv  Detail & Related papers  (2021-12-13T14:26:15Z)
• Remote Multilinear Compressive Learning with Adaptive Compression [107.87219371697063]
  MultiIoT Compressive Learning (MCL) is an efficient signal acquisition and learning paradigm for multidimensional signals. We propose a novel optimization scheme that enables such a feature for MCL models.
  arXiv  Detail & Related papers  (2021-09-02T19:24:03Z)
• Towards Compact CNNs via Collaborative Compression [166.86915086497433]
  We propose a Collaborative Compression scheme, which joints channel pruning and tensor decomposition to compress CNN models. We achieve 52.9% FLOPs reduction by removing 48.4% parameters on ResNet-50 with only a Top-1 accuracy drop of 0.56% on ImageNet 2012.
  arXiv  Detail & Related papers  (2021-05-24T12:07:38Z)
• An Efficient Statistical-based Gradient Compression Technique for Distributed Training Systems [77.88178159830905]
  Sparsity-Inducing Distribution-based Compression (SIDCo) is a threshold-based sparsification scheme that enjoys similar threshold estimation quality to deep gradient compression (DGC) Our evaluation shows SIDCo speeds up training by up to 41:7%, 7:6%, and 1:9% compared to the no-compression baseline, Topk, and DGC compressors, respectively.
  arXiv  Detail & Related papers  (2021-01-26T13:06:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.