Guard: Scalable Straggler Detection and Node Health Management for Large-Scale Training

• URL: http://arxiv.org/abs/2605.17879v1
• Date: Mon, 18 May 2026 05:40:52 GMT
• Title: Guard: Scalable Straggler Detection and Node Health Management for Large-Scale Training
• Abstract summary: Guard is a system for detecting stragglers and ensuring node health in large-scale training clusters.<n>It improves mean FLOPs utilization by up to 1.7x and reduces run-to-run training step variance from 20% to 1%.
• Score: 29.310201112933047
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Training frontier-scale foundation models involves coordinating tens of thousands of GPUs over multi-month runs, where even minor performance degradations can accumulate into substantial efficiency losses. Existing health-check mechanisms, such as NCCL tests or GPU burn-in, primarily focus on functional correctness and often fail to detect fail-slow behaviors that silently degrade system performance. In this paper, we present Guard, a scalable system for detecting stragglers and ensuring node health in large-scale training clusters. Guard combines lightweight online performance monitoring during training with an offline node-sweep mechanism that systematically evaluates and qualifies nodes before they participate in production workloads. This design enables Guard to detect both acute failures and long-running fail-slow behaviors that traditional diagnostics cannot capture. Deployed on large-scale foundation model pretraining workloads, Guard improves mean FLOPs utilization by up to 1.7x, reduces run-to-run training step variance from 20% to 1%, increases mean time to failure (MTTF), and significantly reduces operational and debugging overhead. These results demonstrate that proactive straggler detection and systematic node qualification are critical for maintaining stable and efficient large-scale training.

Related papers

• Mechanism-Driven Monitors for Preemptive Detection of LLM Training Instability [20.49786007379445]
  We study mechanism-driven detection of training instability by deriving internal monitors from the functional role of each critical module.<n>Our fault-injection experiments on low-precision attention, large learning-rate, and combined faults show that these signals provide distinct signatures for different failures.
  arXiv  Detail & Related papers  (2026-06-26T14:18:22Z)
• TimeGuard: Channel-wise Pool Training for Backdoor Defense in Time Series Forecasting [69.01242632976161]
  Time Series Forecasting (TSF) is highly vulnerable to backdoor attacks.<n>Data entanglement induces channel-level signal dilution, rendering sample-filtering and trigger-synthesis defenses ineffective at localizing backdoors.<n>We propose a training-time backdoor defense for TSF, termed TimeGuard.
  arXiv  Detail & Related papers  (2026-05-21T11:58:46Z)
• When Scaling Fails: Network and Fabric Effects on Distributed GPU Training Performance [0.0]
  This paper examines why scaling fails in real systems, with a focus on the role of network and fabric effects.<n>We present an empirical study of distributed GPU training performance across multiple production-scale clusters.<n>We identify recurring failure modes that emerge as training transitions from single-node to multi-node execution.
  arXiv  Detail & Related papers  (2026-02-16T16:19:47Z)
• FreeGAD: A Training-Free yet Effective Approach for Graph Anomaly Detection [54.576802512108685]
  Graph Anomaly Detection (GAD) aims to identify nodes that deviate from the majority within a graph.<n>Existing approaches often suffer from high deployment costs and poor scalability due to their complex and resource-intensive training processes.<n>We propose FreeGAD, a novel training-free yet effective GAD method.
  arXiv  Detail & Related papers  (2025-08-14T12:37:20Z)
• AmorLIP: Efficient Language-Image Pretraining via Amortization [52.533088120633785]
  Contrastive Language-Image Pretraining (CLIP) has demonstrated strong zero-shot performance across diverse downstream text-image tasks.<n>We propose AmorLIP, an efficient CLIP pretraining framework that amortizes expensive computations involved in contrastive learning through lightweight neural networks.
  arXiv  Detail & Related papers  (2025-05-25T05:30:37Z)
• SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Training [60.9776082805359]
  Large Language Models (LLMs) have demonstrated exceptional performance across diverse tasks, yet their training remains highly resource-intensive and susceptible to training instability.<n>This paper presents a comprehensive investigation into gradient spikes observed during LLM training, revealing their prevalence across multiple architectures and datasets.<n>We propose Spike-Aware Adam with Momentum Reset, a novel designed to counteract gradient spikes through momentum reset and spike-aware clipping.
  arXiv  Detail & Related papers  (2025-01-12T15:21:22Z)
• CE-SSL: Computation-Efficient Semi-Supervised Learning for ECG-based Cardiovascular Diseases Detection [16.34314710823127]
  We propose a computation-efficient semi-supervised learning paradigm (CE-SSL) for robust and computation-efficient CVDs detection using ECG. It enables a robust adaptation of pre-trained models on downstream datasets with limited supervision and high computational efficiency. CE-SSL not only outperforms the state-of-the-art methods in multi-label CVDs detection but also consumes fewer GPU footprints, training time, and parameter storage space.
  arXiv  Detail & Related papers  (2024-06-20T14:45:13Z)
• Growing Q-Networks: Solving Continuous Control Tasks with Adaptive Control Resolution [51.83951489847344]
  In robotics applications, smooth control signals are commonly preferred to reduce system wear and energy efficiency. In this work, we aim to bridge this performance gap by growing discrete action spaces from coarse to fine control resolution. Our work indicates that an adaptive control resolution in combination with value decomposition yields simple critic-only algorithms that yield surprisingly strong performance on continuous control tasks.
  arXiv  Detail & Related papers  (2024-04-05T17:58:37Z)
• TRANSOM: An Efficient Fault-Tolerant System for Training LLMs [7.831906758749453]
  Large language models (LLMs) with hundreds of billions or trillions of parameters, represented by chatGPT, have achieved profound impact on various fields. Training LLMs with super-large-scale parameters requires large high-performance GPU clusters and long training periods lasting for months. To address these issues, we propose TRANSOM, a novel fault-tolerant LLM training system.
  arXiv  Detail & Related papers  (2023-10-16T04:06:52Z)
• Online Convolutional Re-parameterization [51.97831675242173]
  We present online convolutional re- parameterization (OREPA), a two-stage pipeline, aiming to reduce the huge training overhead by squeezing the complex training-time block into a single convolution. Compared with the state-of-the-art re-param models, OREPA is able to save the training-time memory cost by about 70% and accelerate the training speed by around 2x. We also conduct experiments on object detection and semantic segmentation and show consistent improvements on the downstream tasks.
  arXiv  Detail & Related papers  (2022-04-02T09:50:19Z)
• Aspis: A Robust Detection System for Distributed Learning [13.90938823562779]
  Machine learning systems can be compromised when some of the computing devices exhibit abnormal (Byzantine) behavior. Our proposed method Aspis assigns gradient computations to worker nodes using a subset-based assignment. We prove the Byzantine resilience and detection guarantees of Aspis under weak and strong attacks and extensively evaluate the system on various large-scale training scenarios.
  arXiv  Detail & Related papers  (2021-08-05T07:24:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.