A Readiness-Driven Runtime for Pipeline-Parallel Training under Runtime Variability

• URL: http://arxiv.org/abs/2605.18750v1
• Date: Mon, 18 May 2026 17:59:18 GMT
• Title: A Readiness-Driven Runtime for Pipeline-Parallel Training under Runtime Variability
• Abstract summary: Pipeline is a key technique for scaling large-model training, but modern workloads exhibit runtime variability in computation and communication.<n>We present-Readiness-First Pipeline (R RFP), a readiness-driven runtime-parallel training.<n>R RFP changes how schedules are consumed at runtime: instead of treating a schedule as a sequence that stages must wait to follow, it treats the schedule as a non-binding hint order for ranking currently ready work.
• Score: 12.676135710714812
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Pipeline parallelism is a key technique for scaling large-model training, but modern workloads exhibit runtime variability in computation and communication. Existing pipeline systems typically consume static, profiled, or adaptively generated schedules as pre-committed execution orders. When realized task readiness diverges from the pre-committed order, stages may wait for not-yet-ready work even though other executable work is available, creating stage misalignment, idle bubbles, and reduced utilization. We present Runtime-Readiness-First Pipeline (RRFP), a readiness-driven runtime for pipeline-parallel training. RRFP changes how schedules are consumed at runtime: instead of treating a schedule as a sequence that stages must wait to follow, it treats the schedule as a non-binding hint order for ranking currently ready work. To support this model, RRFP combines message-driven asynchronous communication, lightweight tensor-parallel coordination for collective consistency, and ready-set arbitration for low-overhead dispatch. We implement RRFP in a Megatron-based training framework and evaluate it on language-only and multimodal workloads at up to 128 GPUs. RRFP improves over fixed-order pipeline baselines across all settings. Using the BFW hint, RRFP achieves up to 1.77$\times$ speedup on language-only workloads and up to 2.77$\times$ on multimodal workloads. In cross-framework comparisons, RRFP with the default BF hint outperforms the faster available external system by up to 1.84$\times$ while preserving training correctness.

Related papers

• Substrate-Portable Execution for Production LLM Workflows [0.0]
  Execution semantics are often coupled to one runtime.<n>Rufus supports real-time serving, asynchronous background tasks, and high-volume batch workloads.<n>We present a binding-adaptive agent execution platform that separates workflow definition from execution substrate.
  arXiv  Detail & Related papers  (2026-09-05T14:52:35Z)
• BASP: Communication-Efficient Batch-Aware Sequence Parallelism for LLM Training [0.48127184936824546]
  Batch- Aware Sequence Parallelism (BASP) is a sequence parallelism approach that leverages batch structure to reduce communication overhead.<n>BASP exploits batch structure by partitioning into disjoint sequence-parallel groups according to the micro- batch size.<n> Experimental results on an NVIDIA A100 cluster show that BASP improves end-to-end training time by up to 1.17 - 1.31x.
  arXiv  Detail & Related papers  (2026-09-02T20:33:23Z)
• GF-DiT: Scheduling Parallelism for Diffusion Transformer Serving [36.55701690878634]
  GF-DiT is a policy-programmable runtime for elastic Diffusion Transformers (DiTs)<n>It adapts the parallelism of running requests according to workload demands and service objectives.<n>It improves throughput by up to 6.01$times$, reduces mean latency by up to 95%, SLO violation rates by up to 90%, and reduces communication-group setup overhead from 778 ms to approximately 60 $$s.
  arXiv  Detail & Related papers  (2026-06-11T15:51:10Z)
• Breaking the Bubble: Asynchronous Pipeline Parallel Training with Bounded Weight Inconsistency [5.093144724934585]
  PACI is a bubble-free pipeline method that bounds forward/backward version drift without weight stashing, prediction, additional parameter copies, or global synchronization.<n>In GPT-style language-model pretraining, PACI matches the stability and final perplexity of synchronous 1F1B-flush, retains the same peak memory footprint, and achieves fully utilized pipeline throughput.<n>These results show that forward/backward inconsistency need not be eliminated: when explicitly bounded, it can be safely traded for substantial efficiency gains.
  arXiv  Detail & Related papers  (2026-06-05T22:33:57Z)
• Demystifying Pipeline Parallelism: First Theory for PipeDream [53.657104889705856]
  In this paper we introduce Randomized PipeDream (PD) as a stale block-SGD abstraction that yields clean non convergence for a PD-style method.<n>We prove that the delay induced by steady-state PD grows as $S2 - S/2 + O(1)$ for $S$ stages, so the stale-read contribution scales as $(2 S4)$, equivalently $(S4/K)$ in the tuned-rate form.
  arXiv  Detail & Related papers  (2026-06-02T11:14:57Z)
• DiscreteRTC: Discrete Diffusion Policies are Natural Asynchronous Executors [57.944744187489185]
  We propose DiscreteRTC, which replaces external corrections with native unmasking.<n>DiscreteRTC is simpler to implement with 0 lines of code for async inpainting, faster at inference with only 0.7x computation compared with generating actions from scratch, and better at execution with 50% higher success rate in real-world dynamic pick task compared with flow-based RTC.
  arXiv  Detail & Related papers  (2026-04-27T23:04:03Z)
• AsyncMesh: Fully Asynchronous Optimization for Data and Pipeline Parallelism [54.8494905524997]
  We introduce asynchronous updates across both parallelism axes, relaxing the co-location requirement.<n>We provide convergence guarantees for both sparse averaging and asynchronous updates.<n>Experiments on large-scale language models demonstrate that our approach matches the performance of the fully synchronous baseline.
  arXiv  Detail & Related papers  (2026-01-30T01:24:47Z)
• Revisiting Parameter Server in LLM Post-Training [16.048510673797523]
  We propose textbfOn-Demand Communication (ODC), which adapts PS into Fully Sharded Data Parallel (FSDP)<n>Compared to FSDP, ODC reduces the synchronization barrier from once per layer to once per minibatch.<n>ODC consistently improves device utilization and training throughput, achieving up to a 36% speedup over standard FSDP.
  arXiv  Detail & Related papers  (2026-01-27T08:44:46Z)
• AdaPtis: Reducing Pipeline Bubbles with Adaptive Pipeline Parallelism on Heterogeneous Models [59.7059443712562]
  AdaPtis is a training system for large language models (LLMs) that supports adaptive pipeline parallelism.<n>Extensive experiments show that AdaPtis achieves an average speedup of 1.42x (up to 2.14x) over Megatron-LM I-1F1B.
  arXiv  Detail & Related papers  (2025-09-28T08:05:13Z)
• Data-Centric Elastic Pipeline Parallelism for Efficient Long-Context LLM Training [40.67232484556671]
  Elastic Pipeline Parallelism (EPP) orchestrates token-level PP and batch-level PP to adapt to resource and workload heterogeneity.<n>InfiniPipe achieves a 1.69x speedup over state-of-the-art systems.
  arXiv  Detail & Related papers  (2025-09-25T15:01:25Z)
• Hierarchical Balance Packing: Towards Efficient Supervised Fine-tuning for Long-Context LLM [49.2709992932292]
  Training Long-Context Large Language Models (LLMs) is challenging, as hybrid training with long-context and short-context data often leads to workload imbalances.<n>Existing works mainly use data packing to alleviate this issue, but fail to consider imbalanced attention computation and wasted communication overhead.<n>This paper proposes Hierarchical Balance Packing (HBP), which designs a novel batch-construction method and training recipe to address those inefficiencies.
  arXiv  Detail & Related papers  (2025-03-10T10:52:50Z)
• SkipPipe: Partial and Reordered Pipelining Framework for Training LLMs in Heterogeneous Networks [0.1638581561083717]
  SkipPipe is a partial pipeline framework to reduce the end-to-end training time for Large Language Models.<n>Our results show that SkipPipe reduces training time by up to $55%$ compared to full pipeline.
  arXiv  Detail & Related papers  (2025-02-27T09:34:23Z)
• Scaling Deep Learning Training with MPMD Pipeline Parallelism [0.5817641705019472]
  JaxPP is a system for efficiently scaling the training of large deep learning models with flexible pipeline parallelism.<n>We introduce a seamless programming model that allows implementing user-defined pipeline schedules for gradient accumulation.<n>JaxPP automatically distributes tasks, corresponding to pipeline stages, over a cluster of nodes and automatically infers the communication among them.
  arXiv  Detail & Related papers  (2024-12-18T22:15:11Z)
• TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models [60.23234205219347]
  TeraPipe is a high-performance token-level pipeline parallel algorithm for synchronous model-parallel training of Transformer-based language models. We show that TeraPipe can speed up the training by 5.0x for the largest GPT-3 model with 175 billion parameters on an AWS cluster.
  arXiv  Detail & Related papers  (2021-02-16T07:34:32Z)
• Parameter-Efficient Transfer Learning with Diff Pruning [108.03864629388404]
  diff pruning is a simple approach to enable parameter-efficient transfer learning within the pretrain-finetune framework. We find that models finetuned with diff pruning can match the performance of fully finetuned baselines on the GLUE benchmark.
  arXiv  Detail & Related papers  (2020-12-14T12:34:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.