Distributed Training using an Intelligent Network

• URL: http://arxiv.org/abs/2608.26453v1
• Date: Wed, 26 Aug 2026 22:58:24 GMT
• Title: Distributed Training using an Intelligent Network
• Abstract summary: Distributed training across a wide area network (WAN) is challenging, as continuous parameter exchange by islands of compute is constrained by limited bandwidth, high latency, and uneven topology.<n>We propose making the network an active participant in training. On the systems side, such networks should leverage (i) multicast technology to replicate outbound traffic and (ii) in-line FPGAs to aggregate inbound traffic.
• Score: 0.14323566945483496
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Distributed training across a wide area network (WAN) is challenging, as continuous parameter exchange by islands of compute is constrained by limited bandwidth, high latency, and uneven topology. We propose making the network an active participant in training. On the systems side, such networks should leverage (i) multicast technology to replicate outbound traffic and (ii) in-line FPGAs to aggregate inbound traffic, to ease egress and ingress bottlenecks. These technologies are used for training across workers within a data center, but this paper extends them to the WAN. On the algorithms side, we develop an optimization framework that produces rich synchronization schedules (namely, rotating cliques of islands) around the underlying network topology and these technologies, to maximize information exchange. Finally, we illustrate this on a nine-city topology modeled on the DoubleZero network, a live programmable WAN equipped with both technologies, and show how the optimal schedules shift with the network's capabilities. Together, these can narrow the gap to the gold standard of colocated training.

Related papers

• Unrolled Neural Networks for Constrained Optimization [83.29547301151177]
  Our framework comprises two coupled neural networks that jointly approximate the saddle point of the Lagrangian.<n>We numerically evaluate the framework on mixed-integer quadratic programs and power allocation in wireless networks.
  arXiv  Detail & Related papers  (2026-01-24T03:12:41Z)
• Resource Allocation in Hybrid Radio-Optical IoT Networks using GNN with Multi-task Learning [11.833896722352568]
  This paper addresses the problem of dual-technology scheduling in hybrid Internet of Things (IoT) networks that integrate Optical NeuralOWC and Radio Frequency (RF)<n>We propose a supervised multi-task learning architecture combining a two-stage Graph Embedding with Transformer (DGET) framework.<n>The proposed framework achieves near-optimal scheduling with over 90% classification accuracy, reduces computational complexity, and demonstrates higher robustness under partial channel observability.
  arXiv  Detail & Related papers  (2025-10-29T15:02:28Z)
• CollaPipe: Adaptive Segment-Optimized Pipeline Parallelism for Collaborative LLM Training in Heterogeneous Edge Networks [57.95170323315603]
  We introduce CollaPipe, a distributed learning framework that integrates collaborative pipeline parallelism with federated aggregation to support self-evolving networks.<n>In CollaPipe, the encoder part is adaptively partitioned into variable-sized segments and deployed across mobile devices for pipeline-parallel training, while the decoder is deployed on edge servers to handle generative tasks.<n>To enhance training efficiency, we formulate a joint optimization problem that adaptively allocates model segments, micro-batches, bandwidth, and transmission power.
  arXiv  Detail & Related papers  (2025-09-24T07:54:01Z)
• Distributed Link Sparsification for Scalable Scheduling Using Graph Neural Networks (Journal Version) [50.894272363373126]
  In wireless networks characterized by dense connectivity, the significant signaling overhead generated by distributed link scheduling algorithms can exacerbate issues like congestion, energy consumption, and radio footprint expansion.<n>We propose a distributed link sparsification scheme employing graph neural networks (GNNs) to reduce scheduling overhead for delay-tolerant traffic while maintaining network capacity.<n>A GNN module is trained to adjust contention thresholds for individual links based on traffic statistics and network topology, enabling links to withdraw from scheduling contention when they are unlikely to succeed.
  arXiv  Detail & Related papers  (2025-09-05T18:59:14Z)
• FusionLLM: A Decentralized LLM Training System on Geo-distributed GPUs with Adaptive Compression [55.992528247880685]
  Decentralized training faces significant challenges regarding system design and efficiency. We present FusionLLM, a decentralized training system designed and implemented for training large deep neural networks (DNNs) We show that our system and method can achieve 1.45 - 9.39x speedup compared to baseline methods while ensuring convergence.
  arXiv  Detail & Related papers  (2024-10-16T16:13:19Z)
• A Deep Reinforcement Learning Approach for Adaptive Traffic Routing in Next-gen Networks [1.1586742546971471]
  Next-gen networks require automation and adaptively adjust network configuration based on traffic dynamics. Traditional techniques that decide traffic policies are usually based on hand-crafted programming optimization and algorithms. We develop a deep reinforcement learning (DRL) approach for adaptive traffic routing.
  arXiv  Detail & Related papers  (2024-02-07T01:48:29Z)
• Multi-Flow Transmission in Wireless Interference Networks: A Convergent Graph Learning Approach [9.852567834643292]
  We introduce a novel algorithm called Dual-stage Interference-Aware Multi-flow Optimization of Network Data-signals (DIAMOND) A centralized stage computes the multi-flow transmission strategy using a novel design of graph neural network (GNN) reinforcement learning (RL) routing agent. Then, a distributed stage improves the performance based on a novel design of distributed learning updates.
  arXiv  Detail & Related papers  (2023-03-27T18:49:47Z)
• Efficient Parallel Split Learning over Resource-constrained Wireless Edge Networks [44.37047471448793]
  In this paper, we advocate the integration of edge computing paradigm and parallel split learning (PSL) We propose an innovative PSL framework, namely, efficient parallel split learning (EPSL) to accelerate model training. We show that the proposed EPSL framework significantly decreases the training latency needed to achieve a target accuracy.
  arXiv  Detail & Related papers  (2023-03-26T16:09:48Z)
• Teal: Learning-Accelerated Optimization of WAN Traffic Engineering [68.7863363109948]
  We present Teal, a learning-based TE algorithm that leverages the parallel processing power of GPUs to accelerate TE control. To reduce the problem scale and make learning tractable, Teal employs a multi-agent reinforcement learning (RL) algorithm to independently allocate each traffic demand. Compared with other TE acceleration schemes, Teal satisfies 6--32% more traffic demand and yields 197--625x speedups.
  arXiv  Detail & Related papers  (2022-10-25T04:46:30Z)
• Themis: A Network Bandwidth-Aware Collective Scheduling Policy for Distributed Training of DL Models [2.6599014990168834]
  Distributed training is a solution to reduce training time by splitting the task across multiple NPUs. Themis is a novel collective scheduling scheme that dynamically schedules collectives to balance the communication loads across all dimensions. Our results show that on average, Themis can improve the network BW utilization of single All-Reduce by 1.88x (2.92x max)
  arXiv  Detail & Related papers  (2021-10-09T06:50:04Z)
• LIBRA: Enabling Workload-aware Multi-dimensional Network Topology Optimization for Distributed Training of Large AI Models [6.980277221943408]
  We motivate the design of multi-dimensional networks within machine learning systems as a cost-efficient mechanism to enhance overall network bandwidth. We introduce LIBRA, a framework specifically focused on optimizing multi-dimensional fabric architectures.
  arXiv  Detail & Related papers  (2021-09-24T06:22:28Z)
• Deep Learning for Ultra-Reliable and Low-Latency Communications in 6G Networks [84.2155885234293]
  We first summarize how to apply data-driven supervised deep learning and deep reinforcement learning in URLLC. To address these open problems, we develop a multi-level architecture that enables device intelligence, edge intelligence, and cloud intelligence for URLLC.
  arXiv  Detail & Related papers  (2020-02-22T14:38:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.