SCOUT: Symmetric Consensus Outlier Detection for Failure Localization in LLM Pre-Training

• URL: http://arxiv.org/abs/2608.11034v1
• Date: Tue, 11 Aug 2026 15:12:14 GMT
• Title: SCOUT: Symmetric Consensus Outlier Detection for Failure Localization in LLM Pre-Training
• Abstract summary: In pre-training, synchronization propagates rank-local stalls, slowdowns, and numerical errors into job-wide symptoms, obscuring their origin.<n>We present SCOUT, a unified runtime failure-localization framework built on one design principle: identify outliers through strict-majority consensus among equivalent replicas.<n>SCOUT integrates with PyTorch, TorchTitan, Megatron-Core, and DeepSpeed without training-loop or framework-source modifications.
• Score: 7.018753691717986
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In LLM pre-training, synchronization propagates rank-local stalls, slowdowns, and numerical errors into job-wide symptoms, obscuring their origin. Existing diagnosis often relies on in-process monitors that cannot report after the trainer blocks or terminates, or on post-mortem logs that preserve only synchronized symptoms; offline health tests lose the workload and operating conditions that triggered the failure. We present SCOUT, a unified runtime failure-localization framework built on one design principle: identify outliers through strict-majority consensus among equivalent replicas. SCOUT aligns replica progress, timing, and numerical evidence, then uses its Consensus Collective Communication (C3) abstraction to identify ranks whose compact signatures disagree with their peers. An out-of-band CPU observer remains responsive when training hangs, whereas in-situ replay exercises recurring stragglers and silent data corruption (SDC) beside the live job with its model state, kernels, allocations, communication path, and thermal and memory pressure present. Collective fingerprints expose rank-local protocol divergence. Clean replay coverage certifies checkpoint numerical integrity, preventing recovery from selecting state corrupted by SDC. SCOUT integrates with PyTorch, TorchTitan, Megatron-Core, and DeepSpeed without training-loop or framework-source modifications. SCOUT is open source at https://github.com/LMResiliency/lm-resiliency.

Related papers

• LMSM: LLM Security Framework Inspired by Linux Security Modules [56.93644694627783]
  Large language models (LLMs) are increasingly deployed with layered defenses, yet malicious prompts can still bypass them.<n>We present Language Model Security Modules (LMSM), a security framework that adapts the separation behind Linux Security Modules (LSM) to LLM serving.
  arXiv  Detail & Related papers  (2026-08-26T12:13:05Z)
• TELLER: Non-intrusive Cross-Layer Root-Cause Analysis for LLM Inference [47.62941623025392]
  We present TELLER, a non-intrusive and Log-aware LLM inference Root-cause analysis framework.<n>TELLER collects NVTX/CUPTI traces and service logs without modifying model binaries.<n>It reconstructs per-request call-chain trees and aligns log lines with corresponding execution steps.<n>On top of these representations, TELLER combines numeric candidate localization with a multimodal root-cause model.
  arXiv  Detail & Related papers  (2026-08-03T09:39:11Z)
• PhyAgentOS: A Self-Evolving Operating System for Embodied Agents with Decoupled Cognitive Planning and Physical Execution [49.776611937968]
  We present PhyAgentOS, a foundation delivering scheduling, verification, memory, benchmarking, and safety as system-level services.<n>Its Session-Centered treats a session, not an action, as the minimum unit of scheduling, compatibility, supervised execution, evidence collection, and acceptance.<n>SessionVerifier distinguishes execution termination from semantic task completion via evidence-grounded verdicts of success, failure, or replan.<n> Benchmarking reuses the deployment session and verification path, so results trace to real execution.
  arXiv  Detail & Related papers  (2026-07-18T04:46:53Z)
• DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation [69.8840101036735]
  Speculative decoding accelerates Large Language Model (LLM) inference.<n>Recent parallel drafters efficiently propose long token sequences in a single forward pass, but suffer from rapid acceptance decay due to a lack of inter-token dependencies.<n>We introduce DSpark, a speculative decoding framework that unifies high- throughput parallel generation with adaptive, load-aware verification.
  arXiv  Detail & Related papers  (2026-07-06T14:28:06Z)
• Governed Shared Memory for Multi-Agent LLM Systems [1.2178992475191557]
  This paper formalizes the fleet-memory problem and identifies four foundational failure modes.<n>We define explicit systems-level primitives: scoped retrieval, temporal supersession, provenance tracking, and policy-governed memory propagation.<n>These primitives are implemented in MemClaw, a production multi-tenant memory service, and evaluated via ArgusFleet.
  arXiv  Detail & Related papers  (2026-06-23T13:04:14Z)
• VCT: A Verifiable Transcript System for LLM Conversations [5.3181399750078535]
  Verifiable Conversation Transcript builds non-linear LLM semantic operations into account-level authenticated state transitions.<n>Security analysis demonstrates that, under standard cryptographic assumptions, VCT guarantees the integrity, consistency, verifiable shareability, and non-repudiation of account-level conversation records.
  arXiv  Detail & Related papers  (2026-06-22T08:17:39Z)
• FERA: Uncertainty-Aware Federated Reasoning for Large Language Models [60.52562148874846]
  We study federated reasoning, where a server improves multi-step reasoning by coordinating with heterogeneous clients holding private demonstrations.<n>The key challenge is that client reliability is query-dependent, while the server cannot inspect client data to determine which contributions are trustworthy.<n>We propose Uncertainty-Aware Federated Reasoning (FERA), a training-free framework based on iterative server-client co-refinement.
  arXiv  Detail & Related papers  (2026-05-11T07:04:51Z)
• Compiling Activation Steering into Weights via Null-Space Constraints for Stealthy Backdoors [48.881343993730844]
  Safety-aligned large language models (LLMs) are increasingly deployed in real-world pipelines.<n> adversaries can distribute backdoored checkpoints that behave normally under standard evaluation but jailbreak when a hidden trigger is present.<n>Recent post-hoc weight-editing methods offer an efficient approach to injecting such backdoors.
  arXiv  Detail & Related papers  (2026-04-14T06:48:33Z)
• OrgForge: A Multi-Agent Simulation Framework for Verifiable Synthetic Corporate Corpora [0.0]
  We present OrgForge, an open-source multi-agent simulation framework that enforces a strict physics-cognition boundary.<n>An actor-local clock enforces causal timestamp correctness across all artifact types.<n>Running a N-day simulation, OrgForge produces interleaved Slack threads, tickets, Confluence pages, Git pull requests, and emails, all traceable to a shared, immutable event log.
  arXiv  Detail & Related papers  (2026-03-16T09:02:24Z)
• STAMP: Outlier-Aware Test-Time Adaptation with Stable Memory Replay [76.06127233986663]
  Test-time adaptation (TTA) aims to address the distribution shift between the training and test data with only unlabeled data at test time. This paper pays attention to the problem that conducts both sample recognition and outlier rejection during inference while outliers exist. We propose a new approach called STAble Memory rePlay (STAMP), which performs optimization over a stable memory bank instead of the risky mini-batch.
  arXiv  Detail & Related papers  (2024-07-22T16:25:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.