FreeRide: Harvesting Bubbles in Pipeline Parallelism

• URL: http://arxiv.org/abs/2409.06941v1
• Date: Wed, 11 Sep 2024 01:46:49 GMT
• Title: FreeRide: Harvesting Bubbles in Pipeline Parallelism
• Authors: Jiashu Zhang, Zihan Pan, Molly, Xu, Khuzaima Daudjee, Sihang Liu,
• Abstract summary: Bubbles in pipeline parallelism account for more than 40% of large language model (LLM) training time. We propose FreeRide, a system designed to harvest bubbles in pipeline parallelism for side tasks. FreeRide achieves 7.8% average cost savings with a negligible overhead of about 1% in training LLMs.
• Score: 5.7868651201471994
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The occurrence of bubbles in pipeline parallelism is an inherent limitation that can account for more than 40% of the large language model (LLM) training time and is one of the main reasons for the underutilization of GPU resources in LLM training. Harvesting these bubbles for GPU side tasks can increase resource utilization and reduce training costs but comes with challenges. First, because bubbles are discontinuous with various shapes, programming side tasks becomes difficult while requiring excessive engineering effort. Second, a side task can compete with pipeline training for GPU resources and incur significant overhead. To address these challenges, we propose FreeRide, a system designed to harvest bubbles in pipeline parallelism for side tasks. FreeRide provides programmers with interfaces to implement side tasks easily, manages bubbles and side tasks during pipeline training, and controls access to GPU resources by side tasks to reduce overhead. We demonstrate that FreeRide achieves 7.8% average cost savings with a negligible overhead of about 1% in training LLMs while serving model training, graph analytics, and image processing side tasks.

Related papers

• PipeFill: Using GPUs During Bubbles in Pipeline-parallel LLM Training [8.637147484753948]
  PipeFill fills pipeline bubbles with execution of other pending jobs. Experiments show that PipeFill can increase overall utilization by up to 63% for GPUs used in large-scale LLM training.
  arXiv  Detail & Related papers  (2024-09-23T22:39:05Z)
• E^2-LLM: Efficient and Extreme Length Extension of Large Language Models [74.1254067728251]
  We propose an Efficient and Extreme length extension method for Large Language Models, called E 2 -LLM, with only one training procedure and dramatically reduced cost. Comprehensive experimental results on multiple benchmark datasets demonstrate the effectiveness of our E 2 -LLM on challenging long-context tasks.
  arXiv  Detail & Related papers  (2024-01-13T02:11:20Z)
• FusionAI: Decentralized Training and Deploying LLMs with Massive Consumer-Level GPUs [57.12856172329322]
  We envision a decentralized system unlocking the potential vast untapped consumer-level GPU. This system faces critical challenges, including limited CPU and GPU memory, low network bandwidth, the variability of peer and device heterogeneity.
  arXiv  Detail & Related papers  (2023-09-03T13:27:56Z)
• FAMO: Fast Adaptive Multitask Optimization [48.59232177073481]
  We introduce Fast Adaptive Multitask Optimization FAMO, a dynamic weighting method that decreases task losses in a balanced way. Our results indicate that FAMO achieves comparable or superior performance to state-of-the-art gradient manipulation techniques.
  arXiv  Detail & Related papers  (2023-06-06T15:39:54Z)
• Actively Learning Costly Reward Functions for Reinforcement Learning [56.34005280792013]
  We show that it is possible to train agents in complex real-world environments orders of magnitudes faster. By enabling the application of reinforcement learning methods to new domains, we show that we can find interesting and non-trivial solutions.
  arXiv  Detail & Related papers  (2022-11-23T19:17:20Z)
• Breadth-First Pipeline Parallelism [0.0]
  Breadth-First Pipeline Parallelism lowers training time, cost and memory usage. It combines a high GPU utilization with a small batch size per GPU, and by making use of fully sharded data parallelism.
  arXiv  Detail & Related papers  (2022-11-11T02:00:32Z)
• Job Scheduling in Datacenters using Constraint Controlled RL [0.0]
  We apply Proportional-Integral-Derivative (PID) Lagrangian methods in Deep Reinforcement Learning to job scheduling problem in the green datacenter environment. Experiments demonstrate improved performance compared to scheduling policies without the PID Lagrangian methods.
  arXiv  Detail & Related papers  (2022-11-10T04:43:14Z)
• Online Evolutionary Batch Size Orchestration for Scheduling Deep Learning Workloads in GPU Clusters [10.395955671683245]
  We propose ONES, an ONline Scheduler for elastic batch size orchestration. ONES automatically manages the elasticity of each job based on the training batch size. We show that ONES can outperform the prior deep learning schedulers with a significantly shorter average job completion time.
  arXiv  Detail & Related papers  (2021-08-08T14:20:05Z)
• Large Batch Simulation for Deep Reinforcement Learning [101.01408262583378]
  We accelerate deep reinforcement learning-based training in visually complex 3D environments by two orders of magnitude over prior work. We realize end-to-end training speeds of over 19,000 frames of experience per second on a single and up to 72,000 frames per second on a single eight- GPU machine. By combining batch simulation and performance optimizations, we demonstrate that Point navigation agents can be trained in complex 3D environments on a single GPU in 1.5 days to 97% of the accuracy of agents trained on a prior state-of-the-art system.
  arXiv  Detail & Related papers  (2021-03-12T00:22:50Z)
• EarlyBERT: Efficient BERT Training via Early-bird Lottery Tickets [106.79387235014379]
  EarlyBERT is a general computationally-efficient training algorithm applicable to both pre-training and fine-tuning of large-scale language models. We are the first to identify structured winning tickets in the early stage of BERT training, and use them for efficient training. EarlyBERT easily achieves comparable performance to standard BERT with 3545% less training time.
  arXiv  Detail & Related papers  (2020-12-31T20:38:20Z)
• Taskflow: A Lightweight Parallel and Heterogeneous Task Graph Computing System [12.813275501138193]
  Taskflow aims to streamline the building of parallel and heterogeneous applications using a lightweight task graph-based approach. Our programming model distinguishes itself as a very general class of task graph parallelism with in-graph control flow. We have demonstrated the promising performance of Taskflow in real-world applications.
  arXiv  Detail & Related papers  (2020-04-23T00:21:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.