A Tabular Schedule Abstraction for Communication-Aware Evaluation of Pipeline-Parallel LLM Training

• URL: http://arxiv.org/abs/2605.24006v1
• Date: Tue, 19 May 2026 13:19:51 GMT
• Title: A Tabular Schedule Abstraction for Communication-Aware Evaluation of Pipeline-Parallel LLM Training
• Abstract summary: Pipeline parallelism is a key technique for distributed training of large language models.<n>We compare GPipe, 1F1B, Chimera, and Hanayo in its restricted regime across multiple modeled system configurations.<n>Our results show that schedule rankings are not abstraction-invariant: communication can negate structural advantages suggested by bubble analysis alone.
• Score: 0.8335524653717384
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Pipeline parallelism is a key technique for distributed training of large language models because it reduces per-device parameter and activation memory. However, comparing pipeline schedules is difficult: analytical models expose structural quantities such as bubble ratios, while end-to-end hardware experiments are costly and system-specific. In this work, we introduce a tabular schedule abstraction and a unified multi-abstraction methodology that connects formula-based reasoning, idealized schedule tables, and communication-aware execution simulation. Using this framework, we compare GPipe, 1F1B, Chimera, and Hanayo in its restricted regime across multiple modeled system configurations. Our results show that schedule rankings are not abstraction-invariant: communication can negate structural advantages suggested by bubble analysis alone. Under the assumptions considered here, GPipe and 1F1B are runtime-equivalent, but 1F1B achieves a lower activation-memory peak. Chimera is advantageous mainly at low microbatch counts and in communication-favorable regimes, while Hanayo is effective in its intended restricted operating point but remains sensitive to network bottlenecks. We further study an asymmetric Chimera-style placement, which does not reduce the global peak memory requirement but reveals limited runtime gains in shallow pipelines. Overall, pipeline schedule quality is meaningful only in the context of the modeled execution environment.

Related papers

• PARM: Pipeline-Adapted Reward Model [60.769414637325326]
  Reward models (RMs) are central to aligning large language models (LLMs) with human preferences, powering advanced decoding strategies.<n>While most prior work focuses on single-step generation, real-world applications increasingly adopt multi-stage pipelines.<n>We investigate this through code generation for optimization, constructing a pipeline that integrates reward models into both formulation and solution stages.
  arXiv  Detail & Related papers  (2026-04-20T14:29:08Z)
• HiAP: A Multi-Granular Stochastic Auto-Pruning Framework for Vision Transformers [3.644142828550762]
  Vision Transformers require significant computational resources and memory bandwidth, severely limiting their deployment on edge devices.<n>We propose Hierarchical Auto-Pruning (HiAP), a continuous relaxation framework that discovers optimal sub-networks in a single end-to-end training phase.<n>HiAP addresses both the memory-bound overhead of loading large matrices and the compute-bound mathematical operations.
  arXiv  Detail & Related papers  (2026-03-12T17:45:38Z)
• OptPipe: Memory- and Scheduling-Optimized Pipeline Parallelism for LLM Training [13.814101909348183]
  Pipeline (PP) has become a standard technique for scaling large language model (LLM) training across multiple devices.<n>In this work, we revisit the pipeline scheduling problem from a principled optimization perspective.<n>We formulate scheduling as a constrained optimization problem that jointly accounts for memory capacity, activation reuse, and pipeline bubble minimization.
  arXiv  Detail & Related papers  (2025-10-06T01:06:33Z)
• AdaPtis: Reducing Pipeline Bubbles with Adaptive Pipeline Parallelism on Heterogeneous Models [59.7059443712562]
  AdaPtis is a training system for large language models (LLMs) that supports adaptive pipeline parallelism.<n>Extensive experiments show that AdaPtis achieves an average speedup of 1.42x (up to 2.14x) over Megatron-LM I-1F1B.
  arXiv  Detail & Related papers  (2025-09-28T08:05:13Z)
• PT$^2$-LLM: Post-Training Ternarization for Large Language Models [52.4629647715623]
  Large Language Models (LLMs) have shown impressive capabilities across diverse tasks, but their large memory and compute demands hinder deployment.<n>We propose PT$2$-LLM, a post-training ternarization framework tailored for LLMs.<n>At its core is an Asymmetric Ternary Quantizer equipped with a two-stage refinement pipeline.
  arXiv  Detail & Related papers  (2025-09-27T03:01:48Z)
• CSGO: Generalized Optimization for Cold Start in Wireless Collaborative Edge LLM Systems [62.24576366776727]
  We propose a latency-aware scheduling framework to minimize total inference latency.<n>We show that the proposed method significantly reduces cold-start latency compared to baseline strategies.
  arXiv  Detail & Related papers  (2025-08-15T07:49:22Z)
• Pangu Embedded: An Efficient Dual-system LLM Reasoner with Metacognition [95.54406667705999]
  Pangu Embedded is an efficient Large Language Model (LLM) reasoner developed on Ascend Neural Processing Units (NPUs)<n>It addresses the significant computational costs and inference latency challenges prevalent in existing reasoning-optimized LLMs.<n>It delivers rapid responses and state-of-the-art reasoning quality within a single, unified model architecture.
  arXiv  Detail & Related papers  (2025-05-28T14:03:02Z)
• Pipeline Parallelism with Controllable Memory [6.135123843073223]
  We show that almost all existing pipeline schedules are memory inefficient. We introduce a family of memory efficient building blocks with controllable activation memory. We can achieve almost zero pipeline bubbles while maintaining the same activation memory as 1F1B.
  arXiv  Detail & Related papers  (2024-05-24T08:54:36Z)
• Zero Bubble Pipeline Parallelism [6.7021820542657045]
  Experimental evaluations show that our method outperforms the 1F1B schedule up to 23% in throughput under a similar memory limit. We believe our results mark a major step forward in harnessing the true potential of pipeline parallelism.
  arXiv  Detail & Related papers  (2023-11-30T10:40:34Z)
• Edge Federated Learning Via Unit-Modulus Over-The-Air Computation (Extended Version) [64.76619508293966]
  This paper proposes a unit-modulus over-the-air computation (UM-AirComp) framework to facilitate efficient edge federated learning. It uploads simultaneously local model parameters and updates global model parameters via analog beamforming. We demonstrate the implementation of UM-AirComp in a vehicle-to-everything autonomous driving simulation platform.
  arXiv  Detail & Related papers  (2021-01-28T15:10:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.