GreedySnake: Accelerating SSD-Offloaded LLM Training with Efficient Scheduling and Optimizer Step Overlapping

• URL: http://arxiv.org/abs/2512.17570v1
• Date: Fri, 19 Dec 2025 13:36:31 GMT
• Title: GreedySnake: Accelerating SSD-Offloaded LLM Training with Efficient Scheduling and Optimizer Step Overlapping
• Abstract summary: This paper introduces GreedySnake, a new SSD-offloaded training system that employs vertical scheduling.<n>Compared to existing systems that use horizontal scheduling, GreedySnake achieves higher training throughput with smaller batch sizes.
• Score: 3.3268565774204704
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: SSD-offloaded training offers a practical and promising approach to making LLM training cost-effective. Building on gradient accumulation with micro-batches, this paper introduces GreedySnake, a new SSD-offloaded training system that employs vertical scheduling, which executes all microbatches of a layer before proceeding to the next. Compared to existing systems that use horizontal scheduling (i.e., executing micro-batches sequentially), GreedySnake achieves higher training throughput with smaller batch sizes, bringing the system much closer to the ideal scenario predicted by the roofline model. To further mitigate the I/O bottleneck, GreedySnake overlaps part of the optimization step with the forward pass of the next iteration. Experimental results on A100 GPUs show that GreedySnake achieves saturated training throughput improvements over ZeRO-Infinity: 1.96x on 1 GPU and 1.93x on 4 GPUs for GPT-65B, and 2.53x on 1 GPU for GPT-175B. The code is open-sourced at https://github.com/npz7yyk/GreedySnake

Related papers

• Mixture-of-Parallelisms: Towards Memory-Efficient Training Stack for Mixture-of-Experts Models [67.17268530365189]
  This paper showcases a memory-efficient training stack for Mixture-of-Experts (MoE) models.<n>It combines and specializes various existing and novel parallelism techniques at different layers and stages of the MoE model training pipeline.
  arXiv  Detail & Related papers  (2026-07-02T08:06:57Z)
• TideGS: Scalable Training of Over One Billion 3D Gaussian Splatting Primitives via Out-of-Core Optimization [19.191940595279206]
  Training 3DGS at billion-primitive scale is fundamentally memory-bound.<n>TideGS is an out-of-core training framework that manages parameters across an SSD- CPU- GPU hierarchy.<n>TideGS enables training with over one billion Gaussians on a single 24 GB GPU.
  arXiv  Detail & Related papers  (2026-05-19T17:40:59Z)
• A Few GPUs, A Whole Lotta Scale: Faithful LLM Training Emulation with PrismLLM [47.09947710347623]
  PrismLLM decouples large-scale execution from the need to access large clusters.<n>Experiments on large-scale LLM training workloads show that PrismLLM accurately reproduces performance and memory behavior.
  arXiv  Detail & Related papers  (2026-05-15T04:58:20Z)
• MegaTrain: Full Precision Training of 100B+ Parameter Large Language Models on a Single GPU [25.228229453758132]
  MegaTrain is a memory-centric system that efficiently trains 100B+ parameter large language models on a single GPU.<n>Unlike traditional GPU-centric systems, MegaTrain stores parameters and states in host memory ( CPU memory) and treats GPUs as transient compute engines.
  arXiv  Detail & Related papers  (2026-04-06T18:43:56Z)
• ReSplat: Learning Recurrent Gaussian Splats [98.14472247275512]
  ReSplat is a feed-forward recurrent Gaussian splatting model that iteratively refines 3D Gaussians without explicitly computing gradients.<n>We introduce a compact reconstruction model that operates in a $16 times$ subsampled space, producing $16 times$ fewer Gaussians than previous per-pixel Gaussian models.<n>Our method achieves state-of-the-art performance while significantly reducing the number of Gaussians and improving the rendering speed.
  arXiv  Detail & Related papers  (2025-10-09T17:59:59Z)
• Cost-Efficient LLM Training with Lifetime-Aware Tensor Offloading via GPUDirect Storage [9.106167012987747]
  TERAIO is a framework for GPU memory expansion using low-cost PCIe-based solid-state drives (SSDs)<n>Its design is driven by our observation that the active tensors take only a small fraction (1.7% on average) of allocated GPU memory in each large language iteration training process.<n>We show that TERAIO improves the training performance of various LLMs by 1.47x on average, and achieves 80.7% of the ideal performance.
  arXiv  Detail & Related papers  (2025-06-06T18:57:20Z)
• ZO2: Scalable Zeroth-Order Fine-Tuning for Extremely Large Language Models with Limited GPU Memory [29.245719403159615]
  We propose a novel framework, ZO2, for efficient zeroth-order fine-tuning of LLMs with only limited GPU memory.<n>Our framework supports an innovative low-bit precision approach in AMP mode to streamline data exchanges between the CPU and GPU.
  arXiv  Detail & Related papers  (2025-03-16T21:58:29Z)
• AutoHete: An Automatic and Efficient Heterogeneous Training System for LLMs [68.99086112477565]
  Transformer-based large language models (LLMs) have demonstrated exceptional capabilities in sequence modeling and text generation.<n>Existing heterogeneous training methods significantly expand the scale of trainable models but introduce substantial communication overheads and CPU workloads.<n>We propose AutoHete, an automatic and efficient heterogeneous training system compatible with both single- GPU and multi- GPU environments.
  arXiv  Detail & Related papers  (2025-02-27T14:46:22Z)
• Democratizing AI: Open-source Scalable LLM Training on GPU-based Supercomputers [65.35142508909892]
  We present a novel four-dimensional hybrid parallel algorithm implemented in a highly scalable, portable, open-source framework called AxoNN.<n>We demonstrate fine-tuning of a 405-billion parameter LLM using AxoNN on Frontier.
  arXiv  Detail & Related papers  (2025-02-12T06:05:52Z)
• APOLLO: SGD-like Memory, AdamW-level Performance [61.53444035835778]
  Large language models (LLMs) are notoriously memory-intensive during training.<n>Various memory-efficient Scals have been proposed to reduce memory usage.<n>They face critical challenges: (i) costly SVD operations; (ii) significant performance trade-offs compared to AdamW; and (iii) still substantial memory overhead to maintain competitive performance.
  arXiv  Detail & Related papers  (2024-12-06T18:55:34Z)
• Grass: Compute Efficient Low-Memory LLM Training with Structured Sparse Gradients [24.58231358634904]
  Large language model (LLM) training and finetuning are often bottlenecked by limited GPU memory. We propose Grass (GRAdient Stuctured Sparsification), a novel approach that leverages sparse projections to transform gradients into structured sparse updates.
  arXiv  Detail & Related papers  (2024-06-25T15:50:32Z)
• Hardware-Aware Parallel Prompt Decoding for Memory-Efficient Acceleration of LLM Inference [23.633481089469836]
  Auto-regressive decoding of Large Language Models (LLMs) results in significant overheads in their hardware performance.<n>We propose a novel parallel prompt decoding that requires only $0.0002$% trainable parameters, enabling efficient training on a single A100-40GB GPU in just 16 hours.<n>Our approach demonstrates up to 2.49$times$ speedup and maintains a minimal memory overhead of just $0.0004$%.
  arXiv  Detail & Related papers  (2024-05-28T22:19:30Z)
• GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection [133.45193150403537]
  Training Large Language Models (LLMs) presents significant memory challenges due to the growing size of weights and GPU states. In this work, we propose Gradient Low-Rank Projection (GaLore) as a memory-efficient training strategy. Our 8-bit GaLore further reduces memory by up to 82.5% and total training memory by 63.3%, compared to a BF16 baseline.
  arXiv  Detail & Related papers  (2024-03-06T07:29:57Z)
• Breaking MLPerf Training: A Case Study on Optimizing BERT [9.486916730173661]
  We present novel approaches for fast large-scale training of BERT model. Load balancing is imperative in distributed BERT training since its training are characterized by samples with various lengths. We propose two new ideas, (1) local presorting based on dataset stratification for load balancing and (2) bucket-wise gradient clipping before allreduce.
  arXiv  Detail & Related papers  (2024-02-04T11:12:17Z)
• Adaptive Elastic Training for Sparse Deep Learning on Heterogeneous Multi-GPU Servers [65.60007071024629]
  We show that Adaptive SGD outperforms four state-of-the-art solutions in time-to-accuracy. We show experimentally that Adaptive SGD outperforms four state-of-the-art solutions in time-to-accuracy.
  arXiv  Detail & Related papers  (2021-10-13T20:58:15Z)
• Large-Scale Training System for 100-Million Classification at Alibaba [43.58719630882661]
  extreme classification has become an essential topic for deep learning. It is very challenging to train a deep model with millions of classes due to the memory and explosion in the last output layer. We build a hybrid parallel training framework to make the training process feasible. Second, we propose a novel softmax variation named KNN softmax, which reduces both the GPU memory consumption and computation costs.
  arXiv  Detail & Related papers  (2021-02-09T06:53:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.