TAGC: Optimizing Gradient Communication in Distributed Transformer Training

• URL: http://arxiv.org/abs/2504.05638v1
• Date: Tue, 08 Apr 2025 03:33:39 GMT
• Title: TAGC: Optimizing Gradient Communication in Distributed Transformer Training
• Abstract summary: Transformer-Aware Gradient Compression (TAGC) is an optimized gradient compression algorithm for transformer-based models.<n>We show that TAGC accelerates training by up to 15% compared to the standard Fully Sharded Data Parallel (FSDP) approach.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The increasing complexity of large language models (LLMs) necessitates efficient training strategies to mitigate the high computational costs associated with distributed training. A significant bottleneck in this process is gradient synchronization across multiple GPUs, particularly in the zero-redundancy parallelism mode. In this paper, we introduce Transformer-Aware Gradient Compression (TAGC), an optimized gradient compression algorithm designed specifically for transformer-based models. TAGC extends the lossless homomorphic compression method by adapting it for sharded models and incorporating transformer-specific optimizations, such as layer-selective compression and dynamic sparsification. Our experimental results demonstrate that TAGC accelerates training by up to 15% compared to the standard Fully Sharded Data Parallel (FSDP) approach, with minimal impact on model quality. We integrate TAGC into the PyTorch FSDP framework, the implementation is publicly available at https://github.com/ipolyakov/TAGC.

Related papers

• Adaptive Targeted Dynamic Chunking for Tokenization-Free Hierarchical Model [1.283285810929198]
  Tokenization-free hierarchical models are emerging as a promising alternative to traditional Large Language Models.<n>We propose Adaptive Targeted Dynamic Chunking (ATDC), a novel byte-compression control mechanism.
  arXiv  Detail & Related papers  (2026-05-28T15:26:34Z)
• TACO: Efficient Communication Compression of Intermediate Tensors for Scalable Tensor-Parallel LLM Training [11.439182801360168]
  TACO (Tensor-parallel Adaptive COmmunication compression) is a robust FP8-based framework for compressing TP intermediate tensors.<n>Uses a data-driven reshaping strategy combined with an Adaptive Scale-Hadamard Transform to enable high-fidelity FP8 quantization.<n>Designs a highly fused compression operator to reduce memory traffic and kernel launch overhead, allowing efficient overlap with communication.
  arXiv  Detail & Related papers  (2026-04-27T06:27:31Z)
• Rethinking Autoregressive Models for Lossless Image Compression via Hierarchical Parallelism and Progressive Adaptation [75.58269386927076]
  Autoregressive (AR) models are often dismissed as impractical due to prohibitive computational cost.<n>This work re-thinks this paradigm, introducing a framework built on hierarchical parallelism and progressive adaptation.<n> Experiments on diverse datasets (natural, satellite, medical) validate that our method achieves new state-of-the-art compression.
  arXiv  Detail & Related papers  (2025-11-14T06:27:58Z)
• EDGC: Entropy-driven Dynamic Gradient Compression for Efficient LLM Training [11.898292264147427]
  Training large language models (LLMs) poses significant challenges regarding computational resources and memory capacity.<n>Existing approaches primarily rely on static gradient compression to enhance communication efficiency.<n>We propose an entropy-driven dynamic gradient compression framework called EDGC.
  arXiv  Detail & Related papers  (2025-11-13T14:05:34Z)
• Protocol Models: Scaling Decentralized Training with Communication-Efficient Model Parallelism [59.79227116582264]
  Scaling models has led to significant advancements in deep learning, but training these models in decentralized settings remains challenging.<n>We propose a novel compression algorithm that compresses both forward and backward passes, enabling up to 99% compression with no convergence degradation.
  arXiv  Detail & Related papers  (2025-06-02T02:19:22Z)
• Efficient Token Compression for Vision Transformer with Spatial Information Preserved [59.79302182800274]
  Token compression is essential for reducing the computational and memory requirements of transformer models. We propose an efficient and hardware-compatible token compression method called Prune and Merge.
  arXiv  Detail & Related papers  (2025-03-30T14:23:18Z)
• Choose Your Model Size: Any Compression of Large Language Models Without Re-Computation [10.376875638696504]
  This work presents Any Compression via Iterative Pruning (ACIP), a novel algorithmic approach to determine a compression-performance trade-off.<n>We use an SVD-reparametrization of linear layers and iteratively prune their singular values with a sparsity-inducing penalty.<n>We show that ACIP seamlessly complements common quantization-based compression techniques.
  arXiv  Detail & Related papers  (2025-02-03T18:40:58Z)
• LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategy [59.1298692559785]
  Key-Value ( KV) cache is crucial component in serving transformer-based autoregressive large language models (LLMs) Existing approaches to mitigate this issue include: (1) efficient attention variants integrated in upcycling stages; (2) KV cache compression at test time; and (3) KV cache compression at test time. We propose a low-rank approximation of KV weight matrices, allowing plug-in integration with existing transformer-based LLMs without model retraining. Our method is designed to function without model tuning in upcycling stages or task-specific profiling in test stages.
  arXiv  Detail & Related papers  (2024-10-04T03:10:53Z)
• Accelerating Large Language Model Training with Hybrid GPU-based Compression [3.204387803072905]
  MPI libraries have been proven to reduce message size significantly and leverage interconnect bandwidth. We investigate the efficacy of compression-assisted MPI collectives under the context of distributed Large Language Model (LLM) training.
  arXiv  Detail & Related papers  (2024-09-04T04:05:30Z)
• HyCoT: A Transformer-Based Autoencoder for Hyperspectral Image Compression [6.0163252984457145]
  Hyperspectral Compression Transformer (HyCoT) is a transformer-based autoencoder for pixelwise HSI compression. Experimental results on the HySpecNet-11k dataset demonstrate that HyCoT surpasses the state of the art across various compression ratios by over 1 dB of PSNR.
  arXiv  Detail & Related papers  (2024-08-16T12:27:46Z)
• LoCo: Low-Bit Communication Adaptor for Large-scale Model Training [63.040522637816906]
  Low-bit communication often degrades training quality due to compression information loss.<n>We propose Low-bit Communication Adaptor (LoCo), which compensates local local GPU nodes before, without compromising quality.<n> Experimental results show that across moving large-scale training model frameworks like Megatron-LM and PyTorchs FSDP, LoCo significantly improves compression communication efficiency.
  arXiv  Detail & Related papers  (2024-07-05T13:01:36Z)
• Accelerating Distributed Deep Learning using Lossless Homomorphic Compression [17.654138014999326]
  We introduce a novel compression algorithm that effectively merges worker-level compression with in-network aggregation. We show up to a 6.33$times$ improvement in aggregation throughput and a 3.74$times$ increase in per-iteration training speed.
  arXiv  Detail & Related papers  (2024-02-12T09:57:47Z)
• A Survey on Transformer Compression [84.18094368700379]
  Transformer plays a vital role in the realms of natural language processing (NLP) and computer vision (CV) Model compression methods reduce the memory and computational cost of Transformer. This survey provides a comprehensive review of recent compression methods, with a specific focus on their application to Transformer-based models.
  arXiv  Detail & Related papers  (2024-02-05T12:16:28Z)
• Efficient Contextformer: Spatio-Channel Window Attention for Fast Context Modeling in Learned Image Compression [1.9249287163937978]
  We introduce the Efficient Contextformer (eContextformer) - a transformer-based autoregressive context model for learned image. It fuses patch-wise, checkered, and channel-wise grouping techniques for parallel context modeling. It achieves 145x lower model complexity and 210Cx faster decoding speed, and higher average bit savings on Kodak, CLI, and Tecnick datasets.
  arXiv  Detail & Related papers  (2023-06-25T16:29:51Z)
• L-GreCo: Layerwise-Adaptive Gradient Compression for Efficient and Accurate Deep Learning [24.712888488317816]
  We provide a framework for adapting the degree of compression across the model's layers dynamically during training. Our framework, called L-GreCo, is based on an adaptive algorithm, which automatically picks the optimal compression parameters for model layers.
  arXiv  Detail & Related papers  (2022-10-31T14:37:41Z)
• Communication-Compressed Adaptive Gradient Method for Distributed Nonconvex Optimization [21.81192774458227]
  One of the major bottlenecks is the large communication cost between the central server and the local workers. Our proposed distributed learning framework features an effective gradient gradient compression strategy.
  arXiv  Detail & Related papers  (2021-11-01T04:54:55Z)
• An Efficient Statistical-based Gradient Compression Technique for Distributed Training Systems [77.88178159830905]
  Sparsity-Inducing Distribution-based Compression (SIDCo) is a threshold-based sparsification scheme that enjoys similar threshold estimation quality to deep gradient compression (DGC) Our evaluation shows SIDCo speeds up training by up to 41:7%, 7:6%, and 1:9% compared to the no-compression baseline, Topk, and DGC compressors, respectively.
  arXiv  Detail & Related papers  (2021-01-26T13:06:00Z)
• PowerGossip: Practical Low-Rank Communication Compression in Decentralized Deep Learning [62.440827696638664]
  We introduce a simple algorithm that directly compresses the model differences between neighboring workers. Inspired by the PowerSGD for centralized deep learning, this algorithm uses power steps to maximize the information transferred per bit.
  arXiv  Detail & Related papers  (2020-08-04T09:14:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.