Piper: A Programmable Distributed Training System

• URL: http://arxiv.org/abs/2606.11169v1
• Date: Tue, 09 Jun 2026 17:48:41 GMT
• Title: Piper: A Programmable Distributed Training System
• Abstract summary: We present Piper, a user-controllable distributed training system that decouples the strategy from the runtime implementation.<n>We show that the combined system maintains performance parity on commonly available strategies such as ZeRO.
• Score: 7.183434086052732
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large-scale model training increasingly relies on composing multiple parallelism strategies, such as data, pipeline, and expert parallelism, together with memory-saving optimizations like ZeRO. Deployed systems for foundation model pretraining often rely on human experts to manually design a high-level parallelism strategy then implement the corresponding low-level execution strategy, making it difficult to adapt the system to new strategies. Meanwhile, many general-purpose frameworks are more flexible but their implementations are still tied to a fixed set of common parallelism strategies, making it challenging to integrate state-of-the-art strategies. We present Piper, a user-controllable distributed training system that decouples the strategy from the runtime implementation. Piper allows users to declare a comprehensive distributed training strategy with a small set of model annotations and scheduling directives. Each directive applies a transformation on Piper's intermediate representation (IR), a unified global training DAG that represents all computation and communication. Using this IR, Piper compiles per-device execution plans and executes them with a distributed runtime agnostic to the strategy. We show that the combined system maintains performance parity on commonly available strategies such as ZeRO, while also enabling additional performance and memory efficiency gains through joint scheduling of compute and communication in composed parallelism strategies such as DeepSeek-V3's DualPipe.

Related papers

• Parallelism Strategy Chaining for Fast Training Convergence [12.463381960118852]
  This paper proposes a new training method that introduces online strategy chaining.<n>ConA ranks candidate strategies during training using a surrogate metric built from compute throughput and gradient statistics.<n>In our evaluation with GPT-3 1.3B, BERT-Large, and Llama-3.2-1B, CONA reaches the target validation perplexity 1.4-9.6x faster than state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-09-07T08:55:14Z)
• Piper: Efficient Large-Scale MoE Training via Resource Modeling and Pipelined Hybrid Parallelism [1.0130502379786543]
  Training MoE models on HPC platforms is hindered by large memory footprints, frequent large-scale communication across heterogeneous networks, and severe workload imbalance.<n>We develop a mathematical model that quantifies memory, compute, and communication requirements for MoE configurations under various parallelization schemes.<n>We introduce Piper, a framework that leverages resource modeling to identify efficient training strategies for MoE models on target HPC platforms.
  arXiv  Detail & Related papers  (2026-05-06T15:47:14Z)
• Learning to Share: Selective Memory for Efficient Parallel Agentic Systems [49.78267008828593]
  Agentic systems solve complex tasks by coordinating multiple agents that iteratively reason, invoke tools, and exchange intermediate results.<n>Recent approaches deploy multiple agent teams running in parallel to explore diverse reasoning trajectories.<n>We propose Learning to Share (LTS), a learned shared-memory mechanism for parallel agentic frameworks.
  arXiv  Detail & Related papers  (2026-02-05T18:20:21Z)
• CONCUR: A Framework for Continual Constrained and Unconstrained Routing [79.85419373937765]
  AI tasks differ in complexity and are best addressed with different computation strategies.<n>Most prior methods build the routing framework by training a single model across all strategies.<n>We propose CONCUR, a continual routing framework that supports both constrained and unconstrained routing.
  arXiv  Detail & Related papers  (2025-12-10T07:30:13Z)
• Experience-Guided Adaptation of Inference-Time Reasoning Strategies [49.954515048847874]
  Experience-Guided Reasoner (EGuR) generates tailored strategies at inference time based on accumulated experience.<n>EGuR achieves up to 14% accuracy improvements over the strongest baselines while reducing computational costs by up to 111x.
  arXiv  Detail & Related papers  (2025-11-14T17:45:28Z)
• Automatic Operator-level Parallelism Planning for Distributed Deep Learning -- A Mixed-Integer Programming Approach [6.449961842220686]
  We propose a bi-level solution framework balancing optimality with computational efficiency.<n>Our framework achieves comparable or superior performance, reducing computational bubbles by half under the same memory constraints.<n>Such capabilities position our solution as both a valuable research tool for exploring optimal parallelization strategies and a practical industrial solution for large-scale AI deployment.
  arXiv  Detail & Related papers  (2025-03-12T13:00:29Z)
• An Adaptive Placement and Parallelism Framework for Accelerating RLHF Training [11.749347656959822]
  We propose a flexible model placement framework that offers two general and agile model placement strategies. Our framework provides a simple user interface and guidelines to easily and flexibly configure these strategies in various training scenarios.
  arXiv  Detail & Related papers  (2023-12-19T03:24:55Z)
• UniPT: Universal Parallel Tuning for Transfer Learning with Efficient Parameter and Memory [69.33445217944029]
  PETL is an effective strategy for adapting pre-trained models to downstream domains. Recent PETL works focus on the more valuable memory-efficient characteristic. We propose a new memory-efficient PETL strategy, Universal Parallel Tuning (UniPT)
  arXiv  Detail & Related papers  (2023-08-28T05:38:43Z)
• Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning [54.99749970495241]
  Alpa automates model-parallel training of large deep learning (DL) models. Alpa generates execution plans that unify data, operator, and pipeline parallelism. Unlike specialized systems, Alpa also generalizes to models with heterogeneous architectures and models without manually-designed plans.
  arXiv  Detail & Related papers  (2022-01-28T10:13:35Z)
• Automap: Towards Ergonomic Automated Parallelism for ML Models [2.469997094590327]
  We present the prototype of an automated partitioner that seamlessly integrates into existing compilers and existing user. Our partitioner enables SPMD-style parallelism that encompasses data parallelism and parameter/activation sharding. Through a combination of inductive tactics and search in a platform-independent partitioning IR, automap can recover expert partitioning strategies such as Megatron sharding for transformer layers.
  arXiv  Detail & Related papers  (2021-12-06T12:09:38Z)
• Parallel Training of Deep Networks with Local Updates [84.30918922367442]
  Local parallelism is a framework which parallelizes training of individual layers in deep networks by replacing global backpropagation with truncated layer-wise backpropagation. We show results in both vision and language domains across a diverse set of architectures, and find that local parallelism is particularly effective in the high-compute regime.
  arXiv  Detail & Related papers  (2020-12-07T16:38:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.