PHOENIX: Resilient LLM Training with Hot-Swapping via Zero-Overhead Checkpoint

• URL: http://arxiv.org/abs/2607.01646v2
• Date: Mon, 06 Jul 2026 23:06:49 GMT
• Title: PHOENIX: Resilient LLM Training with Hot-Swapping via Zero-Overhead Checkpoint
• Abstract summary: We present PHOENIX, a fault-tolerance mechanism that restores large language model (LLM) training via hot-swapping.<n>We show that PHOENIX simultaneously achieves both zero-overhead error-free execution and extremely low recovery cost.
• Score: 11.750747594417339
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: State-of-the-art large language model (LLM) training takes tens of thousands of graphics processing units (GPUs) for months and encounters failures across the software and hardware stack. Existing fault-tolerance mechanisms either impose non-trivial overhead during failure-free execution or suffer from prolonged recovery latency, particularly under scenarios where a small subset of compute nodes experience permanent failures. %The tradeoff between failure-free overhead and recovery latency forms a space forms a Pareto frontier We present PHOENIX to simultaneously address both optimization objectives. PHOENIX incorporates a fault-tolerance mechanism that restores LLM training via hot-swapping, namely by replacing failed nodes with spare nodes without terminating the complete job. The hot-swapping of PHOENIX is enabled by two ideas: First, it exploits an off-critical-path in-memory checkpointing mechanism for spatial redundancy. Second, it introduces a communicator reconstruction protocol that replaces failed nodes with spare nodes at runtime. PHOENIX efficiently overlaps the in-memory checkpointing with computation, thus introducing zero overhead during error-free execution. Upon permanent node failures, PHOENIX can rebuild memory states with minimal recomputation by leveraging in-memory checkpoints. We evaluate PHOENIX across scales (up to 512 NVIDIA A100 GPUs) and LLMs (up to 65B parameters), and observe zero checkpoint overhead with hot-swapping recovery completing in under 40 seconds. These results show that PHOENIX simultaneously achieves both zero-overhead error-free execution and extremely low recovery cost.

Related papers

• Faster Than Flash: Exploiting Attention Sparsity for Efficient Long-Context Decoding [51.48425605758328]
  We present Faster Flash Decoding (FFD), a novel hardware-algorithm co-design framework designed to break the memory wall in long-context decoding.<n>FFD integrates the selector and computer into a fully fused kernel, replacing external metadata indices with content-aware scanning.<n>It achieves up to 11.6x kernel-level speedup and scaling to 256K context length, with 2.37x end-to-end throughput improvement.
  arXiv  Detail & Related papers  (2026-08-31T15:13:20Z)
• SCOUT: Symmetric Consensus Outlier Detection for Failure Localization in LLM Pre-Training [7.018753691717986]
  In pre-training, synchronization propagates rank-local stalls, slowdowns, and numerical errors into job-wide symptoms, obscuring their origin.<n>We present SCOUT, a unified runtime failure-localization framework built on one design principle: identify outliers through strict-majority consensus among equivalent replicas.<n>SCOUT integrates with PyTorch, TorchTitan, Megatron-Core, and DeepSpeed without training-loop or framework-source modifications.
  arXiv  Detail & Related papers  (2026-08-11T15:12:14Z)
• Resume Means Resume: A Machine-Checked Conformance Contract for Checkpoint, Interrupt, and Resume Semantics in Workflow Persistence Layers [0.0]
  A framework that persists execution state so a run can be interrupted, survive a crash, and continue must decide what a resume means for effects that already happened.<n>Five widely deployed agent workflow frameworks answer differently, none exposes a machine-checkable contract.<n>LangGraph 1.2.9 records a second resume value and never consults it, persists-invalid state silently, and re-executes durably recorded work after a real SIGKILL.
  arXiv  Detail & Related papers  (2026-08-04T15:45:31Z)
• Execution-State Capsules: Graph-Bound Execution-State Checkpoint and Restore for Low-Latency, Small-Batch, On-Device Physical-AI Serving [0.5076419064097734]
  We study the opposite regime: low-latency, small-batch, on-device physical-AI serving.<n>We introduce execution-state capsules, a graph-bound checkpoint and restore mechanism for the complete restorable state.<n>This moves reuse from token-addressed KV fragments to graph-bound execution-state boundaries.
  arXiv  Detail & Related papers  (2026-06-18T17:49:36Z)
• Don't Let a Few Network Failures Slow the Entire AllReduce [7.373533433335005]
  We present the first information-theoretic lower bound on AllReduce completion time under asymmetric network bandwidth.<n>We then design OptCC, a four-stage pipelined AllReduce algorithm that approaches this lower bound.<n> Experiments on SimAI confirm that OptCC closes the gap left by existing fault-tolerant schemes.
  arXiv  Detail & Related papers  (2026-06-01T04:40:41Z)
• Towards Distributed Inference of LLMs on a P2P Network [1.1458853556386797]
  Prefix caching can reduce LLM inference latency by reusing KV caches across requests with shared prompts.<n>We propose a decentralized, prefix-cache-aware routing scheme for peer-to-peer LLM serving.
  arXiv  Detail & Related papers  (2026-05-07T15:40:51Z)
• DexWorldModel: Causal Latent World Modeling towards Automated Learning of Embodied Tasks [54.32016216994156]
  We introduce the Causal Latent World Model (CLWM), which employs DINOv3 features as generative targets to disentangle interaction semantics from visual noise.<n>To overcome memory scaling, CLWM features a Dual-State Test-Time Training (TTT) Memory that guarantees a strict $mathcalO(1)$ footprint for long-horizon tasks.<n>We present EmbodiChain, an online framework that establishes the Efficiency Law by injecting an infinite flow of physics-grounded trajectories during training.
  arXiv  Detail & Related papers  (2026-04-13T03:19:36Z)
• Resource-Efficient Iterative LLM-Based NAS with Feedback Memory [49.44875022114861]
  Neural Architecture Search (NAS) automates network design, but conventional methods demand substantial computational resources.<n>We propose a closed-loop pipeline leveraging large language models (LLMs) to iteratively generate, evaluate, and refine convolutional neural network architectures.
  arXiv  Detail & Related papers  (2026-03-12T16:00:22Z)
• Out of the Memory Barrier: A Highly Memory Efficient Training System for LLMs with Million-Token Contexts [68.79341332280062]
  Training Large Language Models (LLMs) on long contexts is severely constrained by prohibitive GPU memory overhead, not training time.<n>We introduce OOMB, a highly memory-efficient training system that directly confronts this barrier.<n>Our approach employs a chunk-recurrent training framework with on-the-fly activation recomputation, which maintains a constant activation memory footprint.
  arXiv  Detail & Related papers  (2026-02-02T13:52:40Z)
• Training LLMs with Fault Tolerant HSDP on 100,000 GPUs [9.97532556913539]
  synchronous training results in a low efficiency due to frequent failures and long recovery time.<n>We propose Fault Tolerant Hybrid-Shared Data Parallelism (FT-HSDP)<n>FT-HSDP uses data parallel replicas as units of fault tolerance.
  arXiv  Detail & Related papers  (2026-01-30T19:57:36Z)
• Reliable and Resilient Collective Communication Library for LLM Training and Serving [5.400838203617311]
  We present R$2$CCL, a fault-tolerant communication library for multi-NIC hardware.<n>R$2$CCL performs rapid connection migration, bandwidth-aware load redistribution, and resilient collective algorithms to maintain progress under failures.<n>Experiments show that R$2$CCL is highly robust to NIC failures, incurring less than 1% training and less than 3% inference overheads.
  arXiv  Detail & Related papers  (2025-12-31T18:53:11Z)
• All is Not Lost: LLM Recovery without Checkpoints [0.1638581561083717]
  CheckFree is an efficient recovery method where a failing stage is substituted by a weighted average of the closest neighboring stages.<n>CheckFree and CheckFree+ outperform checkpointing and redundant computation in terms of convergence in wall-clock time by over 12%.
  arXiv  Detail & Related papers  (2025-06-18T13:48:33Z)
• Rectified Sparse Attention [61.7702154360081]
  Efficient long-sequence generation is a critical challenge for Large Language Models.<n>We propose Rectified Sparse Attention (ReSA), a simple yet effective method that combines block-sparse attention with periodic dense rectification.<n> Experiments across math reasoning, language modeling, and retrieval tasks demonstrate that ReSA achieves near-lossless generation quality.
  arXiv  Detail & Related papers  (2025-06-04T16:01:48Z)
• Flexiffusion: Training-Free Segment-Wise Neural Architecture Search for Efficient Diffusion Models [50.260693393896716]
  Diffusion models (DMs) are powerful generative models capable of producing high-fidelity images but constrained by high computational costs.<n>We propose Flexiffusion, a training-free NAS framework that jointly optimize generation schedules and model architectures without modifying pre-trained parameters.<n>Our work pioneers a resource-efficient paradigm for searching high-speed DMs without sacrificing quality.
  arXiv  Detail & Related papers  (2025-06-03T06:02:50Z)
• vTensor: Flexible Virtual Tensor Management for Efficient LLM Serving [53.972175896814505]
  Large Language Models (LLMs) are widely used across various domains, processing millions of daily requests. Large Language Models (LLMs) are widely used across various domains, processing millions of daily requests.
  arXiv  Detail & Related papers  (2024-07-22T14:37:58Z)
• ReCycle: Resilient Training of Large DNNs using Pipeline Adaptation [2.0181279529015925]
  ReCycle is a system designed for efficient training in the presence of failures. It exploits the inherent functional redundancy in distributed training systems. We show it achieves high training throughput under multiple failures.
  arXiv  Detail & Related papers  (2024-05-22T21:35:56Z)
• AdaPool: Exponential Adaptive Pooling for Information-Retaining Downsampling [82.08631594071656]
  Pooling layers are essential building blocks of Convolutional Neural Networks (CNNs) We propose an adaptive and exponentially weighted pooling method named adaPool. We demonstrate how adaPool improves the preservation of detail through a range of tasks including image and video classification and object detection.
  arXiv  Detail & Related papers  (2021-11-01T08:50:37Z)
• Refining activation downsampling with SoftPool [74.1840492087968]
  Convolutional Neural Networks (CNNs) use pooling to decrease the size of activation maps. We propose SoftPool: a fast and efficient method for exponentially weighted activation downsampling. We show that SoftPool can retain more information in the reduced activation maps.
  arXiv  Detail & Related papers  (2021-01-02T12:09:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.