ZipCCL: Efficient Lossless Data Compression of Communication Collectives for Accelerating LLM Training

• URL: http://arxiv.org/abs/2604.27844v1
• Date: Thu, 30 Apr 2026 13:29:59 GMT
• Title: ZipCCL: Efficient Lossless Data Compression of Communication Collectives for Accelerating LLM Training
• Abstract summary: Communication has emerged as a critical bottleneck in the distributed training of large language models.<n>We introduce ZipCCL, a compressed communication library of collectives for training.<n>ZipCCL reduces communication time by up to 1.35$times$ and achieves end-to-end training speedups of up to 1.18$times$ without any impact on model quality.
• Score: 21.144281084668346
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Communication has emerged as a critical bottleneck in the distributed training of large language models (LLMs). While numerous approaches have been proposed to reduce communication overhead, the potential of lossless compression has remained largely underexplored since compression and decompression typically consume larger overheads than the benefits of reduced communication traffic. We observe that the communication data, including activations, gradients and parameters, during training often follows a near-Gaussian distribution, which is a key feature for data compression. Thus, we introduce ZipCCL, a lossless compressed communication library of collectives for LLM training. ZipCCL is equipped with our novel techniques: (1) theoretically grounded exponent coding that exploits the Gaussian distribution of LLM tensors to accelerate compression without expensive online statistics, (2) GPU-optimized compression and decompression kernels that carefully design memory access patterns and pipeline using communication-aware data layout, and (3) adaptive communication strategies that dynamically switch collective operations based on workload patterns and system characteristics. Evaluated on a 64-GPU cluster using both mixture-of-experts and dense transformer models, ZipCCL reduces communication time by up to 1.35$\times$ and achieves end-to-end training speedups of up to 1.18$\times$ without any impact on model quality.

Related papers

• UCCL-Zip: Lossless Compression Supercharged GPU Communication [37.95759514379356]
  UCCL-Zip supports both point-to-point (P2P) and collective communication without modifying user-facing APIs or compromising numerical correctness.<n>Uzip-P2P employs a split-send pipeline that exposes transmissible data early and overlaps compression with communication.<n>Uzip-NCCL integrates compression into NCCL's persistent kernel model via fused execution.
  arXiv  Detail & Related papers  (2026-04-19T00:05:36Z)
• EDGC: Entropy-driven Dynamic Gradient Compression for Efficient LLM Training [11.898292264147427]
  Training large language models (LLMs) poses significant challenges regarding computational resources and memory capacity.<n>Existing approaches primarily rely on static gradient compression to enhance communication efficiency.<n>We propose an entropy-driven dynamic gradient compression framework called EDGC.
  arXiv  Detail & Related papers  (2025-11-13T14:05:34Z)
• Accelerating Large Language Model Training with Hybrid GPU-based Compression [3.204387803072905]
  MPI libraries have been proven to reduce message size significantly and leverage interconnect bandwidth. We investigate the efficacy of compression-assisted MPI collectives under the context of distributed Large Language Model (LLM) training.
  arXiv  Detail & Related papers  (2024-09-04T04:05:30Z)
• LoCo: Low-Bit Communication Adaptor for Large-scale Model Training [63.040522637816906]
  Low-bit communication often degrades training quality due to compression information loss.<n>We propose Low-bit Communication Adaptor (LoCo), which compensates local local GPU nodes before, without compromising quality.<n> Experimental results show that across moving large-scale training model frameworks like Megatron-LM and PyTorchs FSDP, LoCo significantly improves compression communication efficiency.
  arXiv  Detail & Related papers  (2024-07-05T13:01:36Z)
• FedComLoc: Communication-Efficient Distributed Training of Sparse and Quantized Models [52.13056951033747]
  Federated Learning (FL) has garnered increasing attention due to its unique characteristic of allowing heterogeneous clients to process their private data locally and interact with a central server.<n>A critical bottleneck in FL is the communication cost.<n>Our work is inspired by the innovative Scaffnew algorithm, which has advanced the reduction of communication complexity in FL.<n>We introduce FedComLoc, integrating practical and effective compression into Scaffnew to further enhance communication efficiency.
  arXiv  Detail & Related papers  (2024-03-14T22:29:59Z)
• LoCoDL: Communication-Efficient Distributed Learning with Local Training and Compression [56.01900711954956]
  We introduce LoCoDL, a communication-efficient algorithm that leverages the two popular and effective techniques of Local training, which reduces the communication frequency, and Compression, in which short bitstreams are sent instead of full-dimensional vectors of floats.<n>LoCoDL provably benefits from local training and compression and enjoys a doubly-accelerated communication complexity, with respect to the condition number of the functions and the model dimension, in the general heterogenous regime with strongly convex functions.
  arXiv  Detail & Related papers  (2024-03-07T09:22:50Z)
• Communication-Efficient Distributed Learning with Local Immediate Error Compensation [95.6828475028581]
  We propose the Local Immediate Error Compensated SGD (LIEC-SGD) optimization algorithm. LIEC-SGD is superior to previous works in either the convergence rate or the communication cost.
  arXiv  Detail & Related papers  (2024-02-19T05:59:09Z)
• Fed-CVLC: Compressing Federated Learning Communications with Variable-Length Codes [54.18186259484828]
  In Federated Learning (FL) paradigm, a parameter server (PS) concurrently communicates with distributed participating clients for model collection, update aggregation, and model distribution over multiple rounds. We show strong evidences that variable-length is beneficial for compression in FL. We present Fed-CVLC (Federated Learning Compression with Variable-Length Codes), which fine-tunes the code length in response to the dynamics of model updates.
  arXiv  Detail & Related papers  (2024-02-06T07:25:21Z)
• 1-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB's Convergence Speed [17.953619054149378]
  We propose a new communication-efficient algorithm, 1-bit LAMB, which supports adaptive layerwise learning rates even when communication is compressed. For BERT-Large pre-training task with batch sizes from 8K to 64K, our evaluations demonstrate that 1-bit LAMB with NCCL-based backend is able to achieve up to 4.6x communication volume reduction.
  arXiv  Detail & Related papers  (2021-04-13T10:07:49Z)
• PowerGossip: Practical Low-Rank Communication Compression in Decentralized Deep Learning [62.440827696638664]
  We introduce a simple algorithm that directly compresses the model differences between neighboring workers. Inspired by the PowerSGD for centralized deep learning, this algorithm uses power steps to maximize the information transferred per bit.
  arXiv  Detail & Related papers  (2020-08-04T09:14:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.