From Detection to Recovery: Operational Analysis on LLM Pre-training with 504 GPUs

• URL: http://arxiv.org/abs/2605.09370v1
• Date: Sun, 10 May 2026 06:46:06 GMT
• Title: From Detection to Recovery: Operational Analysis on LLM Pre-training with 504 GPUs
• Abstract summary: This technical report presents an empirical analysis of a 63-node NVIDIA B200 production cluster.<n>Five parties (SKT, Upstage, Lablup, NVIDIA Korea, and VAST Data) share a unified monitoring pipeline.<n>This arrangement enabled joint diagnosis of a 60-node-scale storage I/O bottleneck that did not appear at 2-4-node scale.
• Score: 1.2875754718643733
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large-scale AI training is now fundamentally a distributed systems problem, and hardware failures have become routine operating conditions rather than rare exceptions. Public operational evidence from production training clusters, however, remains scarce. This technical report presents an empirical analysis of a 63-node NVIDIA B200 production cluster (504 GPUs), using 55 days of Prometheus time-series data and 73 days of operational logs covering 224 multi-node training sessions. The cluster operates within a cross-organizational environment in which five parties (SKT, Upstage, Lablup, NVIDIA Korea, and VAST Data) share a unified monitoring pipeline. This arrangement enabled joint diagnosis of a 60-node-scale storage I/O bottleneck that did not appear at 2-4-node scale, a production-scale phenomenon no single team could isolate alone. Drawing on a months-long pre-training campaign, we perform three quantitative analyses yielding four findings. First, statistical analysis over 751 Prometheus metrics and 10 XID-identified GPU failures achieves a 10/10 detection rate (2/10 pre-XID) at ~0.84 false positives per day. No single metric is consistently dominant across failure types, motivating a multi-signal detection strategy. Second, profiling 523 checkpoint events along the GPU VRAM to NFS path attributes the "bandwidth paradox" (1.4-10.4% utilization of 200 Gbps RoCE) to saturation of the 128-slot NFS RPC layer. Third, multi-node failure response shows concentrated exclusions (top 3 of 63 nodes account for >50% of all exclusions) and an auto-retry chain success rate of 33.3% over 12 chains (73 attempts), 2.7x the 12.5% manual recovery rate; the median retry interval is 11 min (IQR 10-11). All analyses are grounded in production infrastructure providing session-level workload management, GPU-centric scheduling, and unified observability.

Related papers

• TriCalRAG: A Three-Strategy, Retrieval-Augmented Benchmark for On-Premise LLM-Based Root Cause Analysis in AIOps [1.8136615181861766]
  Cloud-hosted large language models (LLMs) are increasingly used for root cause analysis (RCA) in AIOps pipelines.<n>They introduce data privacy risk, network latency, and per-query cost that scale poorly with production log volumes.<n>We presentRAG, a benchmark evaluating open-weights served locally via vLLM against a classical LSTM-based log anomaly detector.
  arXiv  Detail & Related papers  (2026-09-13T19:52:46Z)
• Long-Range Indirect Control-Flow Prediction in Stripped Binaries via Dual Virtual Hubs and Multi-Task Graph Learning [44.79384874347874]
  We present ICFlowNet, a unified framework for long-range ICF prediction in stripped binaries.<n>ICFlowNet introduces candidate-aware Dual Virtual Hubs, a Global Code Hub and a Global Data Hub.<n>We also develop a leakage-aware, noise-controlled pipeline with package-level splits, function-level mnemonic-hash deduplication, and a clean test protocol.
  arXiv  Detail & Related papers  (2026-09-03T02:17:23Z)
• From Production Traffic to Post-Training: Building a Self-Hosted LLM That Covers the Corporate Request Mix [55.08143827481755]
  We consolidate traffic from over 200 internal applications onto a single model.<n>We train a separate GRPO expert per axis and merge them via two-stage SLERP.<n>The model absorbs 50% of platform traffic, 116M requests per month, at a fraction of the serving cost.
  arXiv  Detail & Related papers  (2026-09-01T17:39:26Z)
• The Low Frequency Trap: Video Language Models Fail at Simple Event Bookkeeping [70.07961857926377]
  We introduce trace-grounded parametric profiling for event counting in three controlled video tasks.<n>Across 2,190 videos, we vary event count N and frequency F while holding rendering fixed.<n>At an 80% reliability threshold, Gemini 3.6 Flash reliably counts persistent state transitions up to 12 events at 0.5 and 1.0 Hz, yet demonstrates no reliable positive-count region for transient blinking events.
  arXiv  Detail & Related papers  (2026-08-06T17:57:06Z)
• Agora: Collective and Permissionless Internet-Scale Pretraining of Large Language Models [50.733497395381164]
  Agora combines bandwidth-efficient pipeline-parallel model sharding over internet-grade links with multi-party, fault-tolerant collective operations.<n>Report culminates in the first demonstration of its kind: Pluralis-8B, an open, permissionless pretraining run of an 8.6B- parameter model on 500B tokens of FineWeb-Edu.
  arXiv  Detail & Related papers  (2026-07-14T23:32:18Z)
• KernelSight-LM: A Kernel-Level LLM Inference Simulator [27.54401355597591]
  We present KernelSight-LM, a fine-grained inference simulator that models token-level execution and produces kernel-level latency breakdowns.<n>It decomposes each serving step into a kernel model with a learned efficiency term, a communication model, and a host-over-head model.<n>Its kernel-level bottleneck breakdowns support hardware/software co-design and capacity planning.
  arXiv  Detail & Related papers  (2026-06-26T19:43:38Z)
• Benchmarking Inductive Biases for Multivariate Time-Series Anomaly Detection with a Robust Multi-View Channel-Graph Detector [8.29991973809832]
  Ten family-representative detectors are evaluated on five datasets.<n>All methods share the same windowing, scoring, hardware, and metric protocols.<n>We introduce ours, an adaptive detector family combining a NOTEARS-constrained directed channel-graph view with optional patch-attention and temporal-association views.
  arXiv  Detail & Related papers  (2026-05-27T07:55:44Z)
• Resource-Efficient Iterative LLM-Based NAS with Feedback Memory [49.44875022114861]
  Neural Architecture Search (NAS) automates network design, but conventional methods demand substantial computational resources.<n>We propose a closed-loop pipeline leveraging large language models (LLMs) to iteratively generate, evaluate, and refine convolutional neural network architectures.
  arXiv  Detail & Related papers  (2026-03-12T16:00:22Z)
• StrikeWatch: Wrist-worn Gait Recognition with Compact Time-series Models on Low-power FPGAs [10.946464973530214]
  Proper gait patterns can lead to injuries, particularly without expert feedback.<n>Wrist-worn wearables offer a more practical and non-intrusive alternative.<n>This paper introduces StrikeWatch, a compact wrist-worn system that performs entirely on-device, real-time gait recognition.
  arXiv  Detail & Related papers  (2025-10-14T20:28:31Z)
• Multilayer GNN for Predictive Maintenance and Clustering in Power Grids [14.557868383315022]
  Unplanned power outages cost the US economy over $150 billion annually.<n>This study introduces a multilayer Graph Neural Network framework to enhance PdM and enable resilience-based substation clustering.
  arXiv  Detail & Related papers  (2025-07-09T21:44:51Z)
• Machine Learning for Consistency Violation Faults Analysis [0.0]
  This study presents a machine learning-based approach for analyzing the impact of consistency violation faults (cvfs) on distributed systems.<n>By computing program transition ranks and their corresponding effects, the proposed method quantifies the influence of cvfs on system behavior.<n> Experimental results demonstrate promising performance, with a test loss of 4.39 and a mean absolute error of 1.5.
  arXiv  Detail & Related papers  (2025-05-20T22:11:43Z)
• Low-cost Embedded Breathing Rate Determination Using 802.15.4z IR-UWB Hardware for Remote Healthcare [2.6066253940276347]
  We propose a convolutional neural network (CNN) specifically adapted to predict breathing rates from ultra-wideband (UWB) channel impulse response (CIR) data.<n>We show it is feasible to deploy the algorithm on an nRF52840 system-on-chip requiring only 46 KB of memory and operating with an inference time of only 192 ms.
  arXiv  Detail & Related papers  (2025-04-03T07:54:25Z)
• PULL: Reactive Log Anomaly Detection Based On Iterative PU Learning [58.85063149619348]
  We propose PULL, an iterative log analysis method for reactive anomaly detection based on estimated failure time windows. Our evaluation shows that PULL consistently outperforms ten benchmark baselines across three different datasets.
  arXiv  Detail & Related papers  (2023-01-25T16:34:43Z)
• Accelerating Training and Inference of Graph Neural Networks with Fast Sampling and Pipelining [58.10436813430554]
  Mini-batch training of graph neural networks (GNNs) requires a lot of computation and data movement. We argue in favor of performing mini-batch training with neighborhood sampling in a distributed multi-GPU environment. We present a sequence of improvements to mitigate these bottlenecks, including a performance-engineered neighborhood sampler. We also conduct an empirical analysis that supports the use of sampling for inference, showing that test accuracies are not materially compromised.
  arXiv  Detail & Related papers  (2021-10-16T02:41:35Z)
• Stable Prediction on Graphs with Agnostic Distribution Shift [105.12836224149633]
  Graph neural networks (GNNs) have been shown to be effective on various graph tasks with randomly separated training and testing data. In real applications, however, the distribution of training graph might be different from that of the test one. We propose a novel stable prediction framework for GNNs, which permits both locally and globally stable learning and prediction on graphs.
  arXiv  Detail & Related papers  (2021-10-08T02:45:47Z)
• Convolutional Neural Networks for Sleep Stage Scoring on a Two-Channel EEG Signal [63.18666008322476]
  Sleep problems are one of the major diseases all over the world. Basic tool used by specialists is the Polysomnogram, which is a collection of different signals recorded during sleep. Specialists have to score the different signals according to one of the standard guidelines.
  arXiv  Detail & Related papers  (2021-03-30T09:59:56Z)
• TadGAN: Time Series Anomaly Detection Using Generative Adversarial Networks [73.01104041298031]
  TadGAN is an unsupervised anomaly detection approach built on Generative Adversarial Networks (GANs) To capture the temporal correlations of time series, we use LSTM Recurrent Neural Networks as base models for Generators and Critics. To demonstrate the performance and generalizability of our approach, we test several anomaly scoring techniques and report the best-suited one.
  arXiv  Detail & Related papers  (2020-09-16T15:52:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.