Collective Communication Profiling of Modern-day Machine Learning Workloads

• URL: http://arxiv.org/abs/2507.07117v1
• Date: Thu, 03 Jul 2025 20:59:36 GMT
• Title: Collective Communication Profiling of Modern-day Machine Learning Workloads
• Abstract summary: Machine Learning jobs involve periodic communication using operations like AllReduce, AllGather, and Broadcast.<n>These operations may create high bandwidth and bursty traffic patterns, leading to network congestion and packet loss.<n>It is imperative to analyze these patterns, which can be helpful in provisioning network resources depending on the type of machine learning workloads.
• Score: 2.2816873176308907
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Machine Learning jobs, carried out on large number of distributed high performance systems, involve periodic communication using operations like AllReduce, AllGather, and Broadcast. These operations may create high bandwidth and bursty traffic patterns, leading to network congestion and packet loss, thus impacting the performance of these jobs. Hence it is imperative to analyze these patterns, which can be helpful in provisioning network resources depending on the type of machine learning workloads. In this poster we carry out extensive analysis of the collective communication behavior seen in a wide variety of models (ex. DeepSeek, GPT, Llama, etc.) To achieve this we instrument Nvidia Collective Communication Library logging functionality for richer context about the collectives and workloads. We adjust configuration parameters that influence collective communication behavior, such as parallelism, number of nodes, and model type. This overview presents and discusses some of the results on the collective communication behavior for the open source DeepSeek V3 inferencing model, which includes operation type and count, transfer sizes per operation, and request size distribution. Our analysis shows that it makes sense to rethink current collective communication frameworks and network topologies so as to accommodate the effect of network anomalies on the mentioned workloads.

Related papers

• Identifying Super Spreaders in Multilayer Networks [0.6990493129893112]
  We introduce a novel approach to identifying super-spreaders in such networks by leveraging graph neural networks.<n>To this end, we construct a dataset by simulating information diffusion across hundreds of networks.<n>Our model, TopSpreadersNetwork, comprises a relationship-agnostic encoder and a custom aggregation layer.
  arXiv  Detail & Related papers  (2025-05-27T10:14:14Z)
• NetFlowGen: Leveraging Generative Pre-training for Network Traffic Dynamics [72.95483148058378]
  We propose to pre-train a general-purpose machine learning model to capture traffic dynamics with only traffic data from NetFlow records.<n>We address challenges such as unifying network feature representations, learning from large unlabeled traffic data volume, and testing on real downstream tasks in DDoS attack detection.
  arXiv  Detail & Related papers  (2024-12-30T00:47:49Z)
• MIETT: Multi-Instance Encrypted Traffic Transformer for Encrypted Traffic Classification [59.96233305733875]
  Classifying traffic is essential for detecting security threats and optimizing network management.<n>We propose a Multi-Instance Encrypted Traffic Transformer (MIETT) to capture both token-level and packet-level relationships.<n>MIETT achieves results across five datasets, demonstrating its effectiveness in classifying encrypted traffic and understanding complex network behaviors.
  arXiv  Detail & Related papers  (2024-12-19T12:52:53Z)
• AI Flow at the Network Edge [58.31090055138711]
  AI Flow is a framework that streamlines the inference process by jointly leveraging the heterogeneous resources available across devices, edge nodes, and cloud servers.<n>This article serves as a position paper for identifying the motivation, challenges, and principles of AI Flow.
  arXiv  Detail & Related papers  (2024-11-19T12:51:17Z)
• NetGPT: Generative Pretrained Transformer for Network Traffic [4.205009931131087]
  Pretrained models for network traffic can utilize large-scale raw data to learn the essential characteristics of network traffic. In this paper, we make the first attempt to provide a generative pretrained model NetGPT for both traffic understanding and generation tasks.
  arXiv  Detail & Related papers  (2023-04-19T09:04:30Z)
• Dynamic Graph Message Passing Networks for Visual Recognition [112.49513303433606]
  Modelling long-range dependencies is critical for scene understanding tasks in computer vision. A fully-connected graph is beneficial for such modelling, but its computational overhead is prohibitive. We propose a dynamic graph message passing network, that significantly reduces the computational complexity.
  arXiv  Detail & Related papers  (2022-09-20T14:41:37Z)
• Communication Efficient Distributed Learning over Wireless Channels [35.90632878033643]
  Vertical distributed learning exploits the local features collected by multiple learning workers to form a better global model. We propose a novel hierarchical distributed learning framework, where each worker separately learns a low-dimensional embedding of their local observed data. We show that the proposed learning framework is able to achieve almost the same model accuracy as the learning model using the concatenation of all the raw outputs from the learning workers.
  arXiv  Detail & Related papers  (2022-09-04T19:41:21Z)
• Bandit Sampling for Multiplex Networks [8.771092194928674]
  We propose an algorithm for scalable learning on multiplex networks with a large number of layers. Online learning algorithm learns how to sample relevant neighboring layers so that only the layers with relevant information are aggregated during training. We present experimental results on both synthetic and real-world scenarios.
  arXiv  Detail & Related papers  (2022-02-08T03:26:34Z)
• Multi-agent Communication with Graph Information Bottleneck under Limited Bandwidth (a position paper) [92.11330289225981]
  In many real-world scenarios, communication can be expensive and the bandwidth of the multi-agent system is subject to certain constraints. Redundant messages who occupy the communication resources can block the transmission of informative messages and thus jeopardize the performance. We propose a novel multi-agent communication module, CommGIB, which effectively compresses the structure information and node information in the communication graph to deal with bandwidth-constrained settings.
  arXiv  Detail & Related papers  (2021-12-20T07:53:44Z)
• Scalable Perception-Action-Communication Loops with Convolutional and Graph Neural Networks [208.15591625749272]
  We present a perception-action-communication loop design using Vision-based Graph Aggregation and Inference (VGAI) Our framework is implemented by a cascade of a convolutional and a graph neural network (CNN / GNN), addressing agent-level visual perception and feature learning. We demonstrate that VGAI yields performance comparable to or better than other decentralized controllers.
  arXiv  Detail & Related papers  (2021-06-24T23:57:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.