HetCCL: Enabling Collective Communication For Mixed-Vendor Heterogeneous Clusters

• URL: http://arxiv.org/abs/2605.31000v1
• Date: Fri, 29 May 2026 08:34:49 GMT
• Title: HetCCL: Enabling Collective Communication For Mixed-Vendor Heterogeneous Clusters
• Abstract summary: HetCCL is a framework that enables heterogeneous collective communication by efficient P2P transport across heterogeneous devices.<n>HetCCL achieves 17-19x higher bandwidth than Gloo in heterogeneous communications, and speeds up end-to-end training by up to 16.9% in the per-step-time.
• Score: 19.14900229073636
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Training Large Language Models (LLMs) on heterogeneous clusters presents significant challenges for collective communication, as hardware from multiple vendors introduces diverse network and computational characteristics. Existing collective communication frameworks (e.g., NCCL, RCCL) designed for homogeneous environments fail to address mixed-hardware setups, while communication libraries with heterogeneous support (e.g., Gloo, OpenMPI) incur heavy overhead in the data path. This paper presents HetCCL, a framework that enables heterogeneous collective communication by efficient P2P transport across heterogeneous devices (e.g., GPUs), eliminating the host-device memory copy overhead while offloading the control to the CPUs. For combining collectives (e.g., AllReduce, ReduceScatter), HetCCL introduces a border-communicator mechanism that achieves vendor independence by using the intrinsic reduction in the combining collectives in vendor collective communication libraries. With efficient heterogeneous P2P transport and portable reduction mechanism, HetCCL proposes a hierarchical topology abstraction for heterogeneous clusters, dissecting collective communication into cluster-level primitives that guarantee optimal cross-cluster data transfer volume and optimal bandwidth utilization. We implement HetCCL with 4 different vendor support and evaluate it in 4 heterogeneous settings with benchmarks and end-to-end LLM tasks. Our evaluation shows that HetCCL achieves 17-19x higher bandwidth than Gloo in heterogeneous communications, and speeds up end-to-end training by up to 16.9% in the per-step-time.

Related papers

• Unifying Local Communications and Local Updates for LLM Pretraining [22.321729036377846]
  Communication-efficient pre-training of LLMs is increasingly important as training draws on compute distributed across clusters.<n>We introduce GASLoC, a novel decentralized pre-training algorithm that generalizes the notion of communication acceleration to allow a practical gossip-based training framework.<n>We demonstrate that GASLoC outperforms state-of-the-art decentralized algorithms in single step per communication setting for a number of topologies.
  arXiv  Detail & Related papers  (2026-06-09T16:40:54Z)
• GoAgent: Group-of-Agents Communication Topology Generation for LLM-based Multi-Agent Systems [55.12339141172908]
  GoAgent is a communication topology generation method that explicitly treats collaborative groups as the atomic units of MAS construction.<n>Experiments on six benchmarks demonstrate the state-of-the-art performance of GoAgent with 93.84% average accuracy while reducing token consumption by about 17%.
  arXiv  Detail & Related papers  (2026-03-20T06:21:32Z)
• Chamfer-Linkage for Hierarchical Agglomerative Clustering [9.30413560778001]
  We propose emphChamfer-linkage, a novel linkage function that measures the distance between clusters using the Chamfer distance.<n>We find that Chamfer-linkage consistently yields higher-quality clusterings than classical linkages.<n>Our results establish Chamfer-linkage as a practical drop-in replacement for classical linkage functions.
  arXiv  Detail & Related papers  (2026-02-11T02:36:14Z)
• HetCCL: Accelerating LLM Training with Heterogeneous GPUs [5.820335672510985]
  HetCCL is a collective communication library that unifies vendor-specific backends and enables RDMA-based communication across GPUs without requiring driver modifications.<n>HetCCL matches NCCL and RCCL performance in homogeneous setups while uniquely scaling in heterogeneous environments.
  arXiv  Detail & Related papers  (2026-01-30T05:31:29Z)
• One-Shot Hierarchical Federated Clustering [51.490181220883905]
  This paper introduces an efficient one-shot hierarchical Federated Clustering framework.<n>It performs client-end distribution exploration and server-end distribution aggregation.<n>It turns out that the complex cluster distributions across clients can be efficiently explored.
  arXiv  Detail & Related papers  (2026-01-10T02:58:33Z)
• CollaPipe: Adaptive Segment-Optimized Pipeline Parallelism for Collaborative LLM Training in Heterogeneous Edge Networks [57.95170323315603]
  We introduce CollaPipe, a distributed learning framework that integrates collaborative pipeline parallelism with federated aggregation to support self-evolving networks.<n>In CollaPipe, the encoder part is adaptively partitioned into variable-sized segments and deployed across mobile devices for pipeline-parallel training, while the decoder is deployed on edge servers to handle generative tasks.<n>To enhance training efficiency, we formulate a joint optimization problem that adaptively allocates model segments, micro-batches, bandwidth, and transmission power.
  arXiv  Detail & Related papers  (2025-09-24T07:54:01Z)
• ClusterFusion: Expanding Operator Fusion Scope for LLM Inference via Cluster-Level Collective Primitive [38.22906887556149]
  Large language model (LLM) decoding suffers from high latency due to fragmented execution across operators.<n>We introduce two cluster-level communication primitives, ClusterGather and ClusterFusion.<n>We design ClusterFusion, an execution framework that schedules communication and jointly to expand operator fusion scope.
  arXiv  Detail & Related papers  (2025-08-26T09:29:23Z)
• Dual-Segment Clustering Strategy for Hierarchical Federated Learning in Heterogeneous Wireless Environments [22.35256018841889]
  Non-independent and identically distributed (Non- IID) data adversely affects federated learning (FL) This paper proposes a novel dual-segment clustering (DSC) strategy that jointly addresses communication and data heterogeneity in FL. The convergence analysis and experimental results show that the DSC strategy can improve the convergence rate of wireless FL.
  arXiv  Detail & Related papers  (2024-05-15T11:46:47Z)
• V2X-PC: Vehicle-to-everything Collaborative Perception via Point Cluster [58.79477191603844]
  We introduce a new message unit, namely point cluster, to represent the scene sparsely with a combination of low-level structure information and high-level semantic information. This framework includes a Point Cluster Packing (PCP) module to keep object feature and manage bandwidth. Experiments on two widely recognized collaborative perception benchmarks showcase the superior performance of our method compared to the previous state-of-the-art approaches.
  arXiv  Detail & Related papers  (2024-03-25T11:24:02Z)
• CLUSTSEG: Clustering for Universal Segmentation [56.58677563046506]
  CLUSTSEG is a general, transformer-based framework for image segmentation. It tackles different image segmentation tasks (i.e., superpixel, semantic, instance, and panoptic) through a unified neural clustering scheme.
  arXiv  Detail & Related papers  (2023-05-03T15:31:16Z)
• Gradient Coding with Dynamic Clustering for Straggler-Tolerant Distributed Learning [55.052517095437]
  gradient descent (GD) is widely employed to parallelize the learning task by distributing the dataset across multiple workers. A significant performance bottleneck for the per-iteration completion time in distributed synchronous GD is $straggling$ workers. Coded distributed techniques have been introduced recently to mitigate stragglers and to speed up GD iterations by assigning redundant computations to workers. We propose a novel dynamic GC scheme, which assigns redundant data to workers to acquire the flexibility to choose from among a set of possible codes depending on the past straggling behavior.
  arXiv  Detail & Related papers  (2021-03-01T18:51:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.