GPUAlert: A Zero-Instrumentation Process-Boundary Monitor for Diagnosing GPU Training-Job Failures

• URL: http://arxiv.org/abs/2607.01409v1
• Date: Wed, 01 Jul 2026 19:15:16 GMT
• Title: GPUAlert: A Zero-Instrumentation Process-Boundary Monitor for Diagnosing GPU Training-Job Failures
• Abstract summary: GPUAlert is a command-line wrapper that monitors any training command at the process boundary.<n>It emails a structured notification on completion carrying a classified failure cause, durable logs, and output artifacts.<n>We release a labelled corpus of 474 GPU training logs across 15 failure classes and a reproducible evaluation harness.
• Score: 34.01019358995093
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: GPU training jobs fail often, roughly two in five on large production clusters, yet the operator typically learns of a failure only by reconnecting hours later. Experiment trackers require editing the training script and maintaining a cloud connection; the scheduler's mail hook delivers a single status line with no cause and no logs. GPUAlert is a command-line wrapper that monitors any training command at the process boundary, and with no change to that command, emails a structured notification on completion carrying a classified failure cause, durable logs, and output artifacts. The tool is organized around three reliability primitives: a pre-launch log guarantee that establishes the durable destination before the child process can crash, notifier isolation that makes the wrapper's exit code a pure function of the child's status regardless of whether the email succeeds, and a non-silent artifact budget that bounds attachment size without ever dropping output silently. We release a labelled corpus of 474 GPU training logs across 15 failure classes and a reproducible evaluation harness. On the twelve hardware-reproduced classes, the ordered-rule classifier reaches 0.997 macro-F1, against 0.830 for unordered keyword matching and 0.133 for exit-code inspection. Wrapper overhead is a constant approximately 3ms per job; the pre-launch guarantee preserves a log where a shell redirect yields nothing; and across all 15 failure modes the wrapper returns the child's exit code unchanged even when the SMTP relay is unreachable.

Related papers

• Privacy-Preserving Topology-Guided Safety for LLM-Based Multi-Agent Systems via Federated Graph Learning [67.04448659688579]
  FGLGuard cuts AgentDojo's ground-success rate by 43% at near-unguarded utility, zero API cost, and negligible capability loss.<n>Federation is not optional: off-the-shelf transfer collapses under distribution shift.<n>Live FGLGuard cuts AgentDojo's ground-success rate by 43% at near-unguarded utility, zero API cost, and negligible capability loss.
  arXiv  Detail & Related papers  (2026-09-02T07:57:12Z)
• LMSM: LLM Security Framework Inspired by Linux Security Modules [56.93644694627783]
  Large language models (LLMs) are increasingly deployed with layered defenses, yet malicious prompts can still bypass them.<n>We present Language Model Security Modules (LMSM), a security framework that adapts the separation behind Linux Security Modules (LSM) to LLM serving.
  arXiv  Detail & Related papers  (2026-08-26T12:13:05Z)
• SCOUT: Symmetric Consensus Outlier Detection for Failure Localization in LLM Pre-Training [7.018753691717986]
  In pre-training, synchronization propagates rank-local stalls, slowdowns, and numerical errors into job-wide symptoms, obscuring their origin.<n>We present SCOUT, a unified runtime failure-localization framework built on one design principle: identify outliers through strict-majority consensus among equivalent replicas.<n>SCOUT integrates with PyTorch, TorchTitan, Megatron-Core, and DeepSpeed without training-loop or framework-source modifications.
  arXiv  Detail & Related papers  (2026-08-11T15:12:14Z)
• Resume Means Resume: A Machine-Checked Conformance Contract for Checkpoint, Interrupt, and Resume Semantics in Workflow Persistence Layers [0.0]
  A framework that persists execution state so a run can be interrupted, survive a crash, and continue must decide what a resume means for effects that already happened.<n>Five widely deployed agent workflow frameworks answer differently, none exposes a machine-checkable contract.<n>LangGraph 1.2.9 records a second resume value and never consults it, persists-invalid state silently, and re-executes durably recorded work after a real SIGKILL.
  arXiv  Detail & Related papers  (2026-08-04T15:45:31Z)
• Fantastic Adaptive Taxonomies and How to Use Them [100.20614173157708]
  AdaMAST converts execution traces into a compact, evidence-grounded failure taxonomy.<n>No code is hand-authored, no trace human-annotated.<n>In agent-system search, taxonomy-coded diagnoses of failed candidates outperform free-form.
  arXiv  Detail & Related papers  (2026-07-17T17:41:16Z)
• Taming the Drift: Context-aware Repair of Dockerfile Drift during Software Evolution [52.50730821321986]
  Cadre is a context-aware framework for automated Dockerfile drift repair.<n>CDG maps each Dockerfile instruction to its file-level and inter-instruction dependencies.<n>DodeX mines real-world Dockerfile drift instances from GitHub Actions CI logs.
  arXiv  Detail & Related papers  (2026-07-14T09:12:26Z)
• Early Diagnosis of Wasted Computation in Multi-Agent LLM Systems via Failure-Aware Observability [8.036549927091286]
  This paper introduces a failure-aware observability framework for diagnosing wasted computation in multi-agent LLM traces.<n>We instantiate the framework in a three- agent question-answering system and evaluate it on 165 GAIA validation traces under identical execution caps.
  arXiv  Detail & Related papers  (2026-05-31T17:50:11Z)
• FAME: Failure-Aware Mixture-of-Experts for Message-Level Log Anomaly Detection [6.975728318257755]
  We present FAME (Failure-Aware Mixture-of-Experts), a label-efficient message-level mixture-of-experts framework.<n>FAME trains a lightweight router and domain experts that run on-premise and output anomaly predictions.<n>On BGL, FAME achieves F1 = 98.16 at K = 100 reducing annotation effort by 76x and detects 86.3% of anomalies from unseen EventIDs.
  arXiv  Detail & Related papers  (2026-05-21T17:34:53Z)
• MemLineage: Lineage-Guided Enforcement for LLM Agent Memory [3.058164432275431]
  We introduce MemLineage, a defense for LLM agent memory that attaches cryptographic provenance and LLM-mediated lineage to every entry.<n>Recent and concurrent work shows that untrusted content can be written into persistent agent state and re-enter later sessions as an instruction.<n>MemLineage treats this as a chain-of-custody problem rather than a filtering problem.
  arXiv  Detail & Related papers  (2026-05-14T06:07:54Z)
• ReCoVer: Resilient LLM Pre-Training System via Fault-Tolerant Collective and Versatile Workload [18.933354857797056]
  Pre-training large language models on massive GPU clusters has made hardware faults routine rather than rare.<n>We propose ReCoVer, a resilient pre-training system that upholds a single invariant.<n>For comparison with checkpoint-and-restart baselines, ReCoVer demonstrates $2.23times$ higher effective throughput after successive failures.
  arXiv  Detail & Related papers  (2026-05-11T20:28:31Z)
• PrefixGuard: From LLM-Agent Traces to Online Failure-Warning Monitors [14.336100401626062]
  Large language model (LLM) agents execute long, tool-using tasks where final outcome checks can arrive too late for intervention.<n>We introduce PrefixGuard, a trace-to-monitor framework with an offline StepView induction step followed by supervised monitor training.<n>Across WebArena, $2$-Bench, SkillsBench, and TerminalBench, the strongest PrefixGuard monitors reach 0.900/0.70.533/0.557 AUPRC.
  arXiv  Detail & Related papers  (2026-05-07T15:49:48Z)
• Compiling Activation Steering into Weights via Null-Space Constraints for Stealthy Backdoors [48.881343993730844]
  Safety-aligned large language models (LLMs) are increasingly deployed in real-world pipelines.<n> adversaries can distribute backdoored checkpoints that behave normally under standard evaluation but jailbreak when a hidden trigger is present.<n>Recent post-hoc weight-editing methods offer an efficient approach to injecting such backdoors.
  arXiv  Detail & Related papers  (2026-04-14T06:48:33Z)
• SemGuard: Real-Time Semantic Evaluator for Correcting LLM-Generated Code [46.20378145112059]
  Post-hoc repair pipelines detect such faults only after execution.<n>We present SemGuard, a semantic-evaluator-driven framework that performs real-time, line-level semantic supervision.
  arXiv  Detail & Related papers  (2025-09-29T09:21:32Z)
• Get my drift? Catching LLM Task Drift with Activation Deltas [55.75645403965326]
  Task drift allows attackers to exfiltrate data or influence the LLM's output for other users.<n>We show that a simple linear classifier can detect drift with near-perfect ROC AUC on an out-of-distribution test set.<n>We observe that this approach generalizes surprisingly well to unseen task domains, such as prompt injections, jailbreaks, and malicious instructions.
  arXiv  Detail & Related papers  (2024-06-02T16:53:21Z)
• Unsupervised Continual Anomaly Detection with Contrastively-learned Prompt [80.43623986759691]
  We introduce a novel Unsupervised Continual Anomaly Detection framework called UCAD. The framework equips the UAD with continual learning capability through contrastively-learned prompts. We conduct comprehensive experiments and set the benchmark on unsupervised continual anomaly detection and segmentation.
  arXiv  Detail & Related papers  (2024-01-02T03:37:11Z)
• Self-Supervised Log Parsing [59.04636530383049]
  Large-scale software systems generate massive volumes of semi-structured log records. Existing approaches rely on log-specifics or manual rule extraction. We propose NuLog that utilizes a self-supervised learning model and formulates the parsing task as masked language modeling.
  arXiv  Detail & Related papers  (2020-03-17T19:25:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.