Parallelism Strategy Chaining for Fast Training Convergence

• URL: http://arxiv.org/abs/2609.07236v1
• Date: Mon, 07 Sep 2026 08:55:14 GMT
• Title: Parallelism Strategy Chaining for Fast Training Convergence
• Abstract summary: This paper proposes a new training method that introduces online strategy chaining.<n>ConA ranks candidate strategies during training using a surrogate metric built from compute throughput and gradient statistics.<n>In our evaluation with GPT-3 1.3B, BERT-Large, and Llama-3.2-1B, CONA reaches the target validation perplexity 1.4-9.6x faster than state-of-the-art methods.
• Score: 12.463381960118852
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Selecting a parallelism strategy - the configuration of data, tensor, and pipeline parallelism degrees together with micro- and global-batch sizes - largely determines the training efficiency of large language models. State-of-the-art methods search for a parallelism strategy offline and select the single strategy that minimizes per-iteration time. But we find that they neglect the target validation perplexity and time-to-perplexity (TTP). In particular, our analysis reveals that the best strategy yielding the fastest perplexity improvement changes multiple times during training. As a result, state-of-the-art methods are 1.8-11.4x slower in TTP than the strategy sequence that selects the best strategy at each iteration. This paper proposes CONA, a new training method that introduces online strategy chaining. Instead of a single strategy selected offline, CONA ranks candidate strategies during training using a surrogate metric built from compute throughput and gradient statistics, and switches the current strategy to a new strategy with a higher metric. In our evaluation with GPT-3 1.3B, BERT-Large, and Llama-3.2-1B, CONA reaches the target validation perplexity 1.4-9.6x faster than state-of-the-art methods. Moreover, CONA closely tracks the perplexity achieved by the sequence that selects the best strategy at each iteration, within 2.6%.

Related papers

• Piper: A Programmable Distributed Training System [7.183434086052732]
  We present Piper, a user-controllable distributed training system that decouples the strategy from the runtime implementation.<n>We show that the combined system maintains performance parity on commonly available strategies such as ZeRO.
  arXiv  Detail & Related papers  (2026-06-09T17:48:41Z)
• StraTA: Incentivizing Agentic Reinforcement Learning with Strategic Trajectory Abstraction [76.50439715778869]
  StraTA is a framework that introduces an explicit trajectory-level strategy into agentic reinforcement learning.<n>Experiments on ALFWorld, WebShop, and SciWorld show that StraTA consistently improves both sample efficiency and final performance.
  arXiv  Detail & Related papers  (2026-05-07T17:51:16Z)
• Strategy Executability in Mathematical Reasoning: Leveraging Human-Model Differences for Effective Guidance [86.46794021499511]
  We show a previously underexplored gap between strategy usage and strategy executability.<n>We propose Selective Strategy Retrieval (SSR), a test-time framework that explicitly models executability.<n> SSR yields reliable and consistent improvements over direct solving, in-context learning, and single-source guidance.
  arXiv  Detail & Related papers  (2026-02-26T03:34:23Z)
• Placement Semantics for Distributed Deep Learning: A Systematic Framework for Analyzing Parallelism Strategies [0.0]
  Training large language models requires distributing computation across many accelerators.<n>No unified systematic framework predicts their behavior.<n>We introduce placement semantics: each strategy is specified by how it places four training states across devices.
  arXiv  Detail & Related papers  (2026-01-05T18:01:38Z)
• Experience-Guided Adaptation of Inference-Time Reasoning Strategies [49.954515048847874]
  Experience-Guided Reasoner (EGuR) generates tailored strategies at inference time based on accumulated experience.<n>EGuR achieves up to 14% accuracy improvements over the strongest baselines while reducing computational costs by up to 111x.
  arXiv  Detail & Related papers  (2025-11-14T17:45:28Z)
• AutoDAN-Reasoning: Enhancing Strategies Exploration based Jailbreak Attacks with Test-Time Scaling [54.47844626555395]
  AutoDAN-Turbo employs a lifelong learning agent to build a rich library of attack strategies from scratch.<n>While highly effective, its test-time generation process involves sampling a strategy and generating a single corresponding attack prompt.<n>We propose to further improve the attack performance of AutoDAN-Turbo through test-time scaling.
  arXiv  Detail & Related papers  (2025-10-06T21:16:09Z)
• Plan before Solving: Problem-Aware Strategy Routing for Mathematical Reasoning with LLMs [49.995906301946]
  Existing methods usually leverage a fixed strategy to guide Large Language Models (LLMs) to perform mathematical reasoning.<n>Our analysis reveals that the single strategy cannot adapt to problem-specific requirements and thus overlooks the trade-off between effectiveness and efficiency.<n>We propose Planning and Routing through Instance-Specific Modeling (PRISM), a novel framework that decouples mathematical reasoning into two stages: strategy planning and targeted execution.
  arXiv  Detail & Related papers  (2025-09-29T07:22:41Z)
• Stratify: Unifying Multi-Step Forecasting Strategies [0.0]
  Stratify is a framework that addresses multi-step forecasting and unifying existing strategies.<n>In over 84% of 1080 experiments, novel strategies in Stratify improved performance compared to all existing ones.<n>Our results are the most comprehensive benchmarking of known and novel forecasting strategies.
  arXiv  Detail & Related papers  (2024-12-29T16:06:46Z)
• PaSE: Parallelization Strategies for Efficient DNN Training [0.09889128046943638]
  Training a deep neural network (DNN) requires substantial computational and memory requirements. Standard practice is to use data parallelism because of its simplicity. Expert-designed strategies have been proposed on a case-by-case basis using domain specific knowledge.
  arXiv  Detail & Related papers  (2024-07-04T15:21:20Z)
• Deep Reinforcement Learning for Exact Combinatorial Optimization: Learning to Branch [13.024115985194932]
  We propose a new approach for solving the data labeling and inference issues in optimization based on the use of the reinforcement learning (RL) paradigm. We use imitation learning to bootstrap an RL agent and then use Proximal Policy (PPO) to further explore global optimal actions.
  arXiv  Detail & Related papers  (2022-06-14T16:35:58Z)
• LAS-AT: Adversarial Training with Learnable Attack Strategy [82.88724890186094]
  "Learnable attack strategy", dubbed LAS-AT, learns to automatically produce attack strategies to improve the model robustness. Our framework is composed of a target network that uses AEs for training to improve robustness and a strategy network that produces attack strategies to control the AE generation.
  arXiv  Detail & Related papers  (2022-03-13T10:21:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.