Mechanism-Driven Monitors for Preemptive Detection of LLM Training Instability

• URL: http://arxiv.org/abs/2606.28116v1
• Date: Fri, 26 Jun 2026 14:18:22 GMT
• Title: Mechanism-Driven Monitors for Preemptive Detection of LLM Training Instability
• Abstract summary: We study mechanism-driven detection of training instability by deriving internal monitors from the functional role of each critical module.<n>Our fault-injection experiments on low-precision attention, large learning-rate, and combined faults show that these signals provide distinct signatures for different failures.
• Score: 20.49786007379445
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Frontier large language model training consumes massive accelerator fleets and long wall-clock computation, making stability failures costly when they occur. After a numerical or a hyperparameter fault has already destabilized the training dynamics, it may continue for thousands of steps while loss and gradient norms still appear normal. We study mechanism-driven detection of training instability by deriving internal monitors from the functional role of each critical module and from the earliest computational sites where failures are expected to produce measurable signatures. For low-precision flash attention, we monitor the spectral entropy of a QK bilinear decomposition, whose first-order term becomes abnormal before the loss fully collapses. For MoE routers, we derive indicators from their role in expert selection. Our fault-injection experiments on low-precision attention, large learning-rate, and combined faults show that these signals provide distinct signatures for different failures, triggering thousands of steps before loss divergence.

Related papers

• Mitigating Error Amplification in Fast Adversarial Training [58.74042726356826]
  Fast Adversarial Training (FAT) has proven effective in enhancing model robustness by encouraging networks to learn perturbation-invariant representations.<n>FAT often suffers from catastrophic overfitting (CO), where the model overfits to the training attack and fails to generalize to unseen ones.<n>We propose a Distribution-aware Dynamic Guidance (DDG) strategy that dynamically adjusts both the perturbation budget and supervision signal.
  arXiv  Detail & Related papers  (2026-04-27T11:23:18Z)
• Causally-Constrained Probabilistic Forecasting for Time-Series Anomaly Detection [1.8995711908407733]
  This study introduces a causally-constrained probabilistic forecasting framework for anomaly detection.<n>It integrates an explicit time-lagged causal graph prior with deep sequence modeling.<n>Experiments indicate that the proposed method achieves state-of-the-art detection performance.
  arXiv  Detail & Related papers  (2026-04-20T09:24:28Z)
• Shifting Uncertainty to Critical Moments: Towards Reliable Uncertainty Quantification for VLA Model [10.781664119473966]
  We propose a unified uncertainty quantification approach for predicting rollout success versus failure.<n>Our method substantially improves failure prediction accuracy and yields more reliable signals for failure detection.
  arXiv  Detail & Related papers  (2026-03-18T22:52:03Z)
• Axle Sensor Fusion for Online Continual Wheel Fault Detection in Wayside Railway Monitoring [1.1199585259018456]
  This work presents a semantic-aware, label-efficient continual learning framework for railway fault diagnostics.<n>The model detects minor imperfections due to flats and polygonization, while adapting to evolving operational conditions.
  arXiv  Detail & Related papers  (2026-02-18T00:14:18Z)
• ProbeLLM: Automating Principled Diagnosis of LLM Failures [89.44131968886184]
  We propose ProbeLLM, a benchmark-agnostic automated probing framework that elevates weakness discovery from individual failures to structured failure modes.<n>By restricting probing to verifiable test cases and leveraging tool-augmented generation and verification, ProbeLLM grounds failure discovery in reliable evidence.
  arXiv  Detail & Related papers  (2026-02-13T14:33:13Z)
• Metacognitive Self-Correction for Multi-Agent System via Prototype-Guided Next-Execution Reconstruction [58.51530390018909]
  Large Language Model based multi-agent systems excel at collaborative problem solving but remain brittle to cascading errors.<n>We present MASC, a metacognitive framework that endows MAS with real-time, unsupervised, step-level error detection and self-correction.
  arXiv  Detail & Related papers  (2025-10-16T05:35:37Z)
• ResAD: Normalized Residual Trajectory Modeling for End-to-End Autonomous Driving [64.42138266293202]
  ResAD is a Normalized Residual Trajectory Modeling framework.<n>It reframes the learning task to predict the residual deviation from an inertial reference.<n>On the NAVSIM benchmark, ResAD achieves a state-of-the-art PDMS of 88.6 using a vanilla diffusion policy.
  arXiv  Detail & Related papers  (2025-10-09T17:59:36Z)
• Spectral Alignment as Predictor of Loss Explosion in Neural Network Training [31.831122738372603]
  Loss explosions in training deep neural networks can nullify multi-million dollar training runs.<n>We introduce Spectral Alignment, a novel metric that monitors the distributional alignment between layer inputs and the principal singular vectors of weight matrices.<n> Empirical results on language models demonstrate that monitoring the SA distribution provides a significantly earlier and clearer warning of loss explosions than traditional scalar metrics.
  arXiv  Detail & Related papers  (2025-10-05T13:35:34Z)
• CVCM Track Circuits Pre-emptive Failure Diagnostics for Predictive Maintenance Using Deep Neural Networks [0.15056924758531146]
  Track circuits are critical for railway operations, acting as main signalling sub-system to locate trains.<n>Many failures originate as subtle anomalies that evolve over time, often not visually apparent in monitored signals.<n>We propose a predictive maintenance framework that classifies anomalies well before they escalate into failures.
  arXiv  Detail & Related papers  (2025-08-12T16:13:51Z)
• Unsupervised Continual Anomaly Detection with Contrastively-learned Prompt [80.43623986759691]
  We introduce a novel Unsupervised Continual Anomaly Detection framework called UCAD. The framework equips the UAD with continual learning capability through contrastively-learned prompts. We conduct comprehensive experiments and set the benchmark on unsupervised continual anomaly detection and segmentation.
  arXiv  Detail & Related papers  (2024-01-02T03:37:11Z)
• Causal Disentanglement Hidden Markov Model for Fault Diagnosis [55.90917958154425]
  We propose a Causal Disentanglement Hidden Markov model (CDHM) to learn the causality in the bearing fault mechanism. Specifically, we make full use of the time-series data and progressively disentangle the vibration signal into fault-relevant and fault-irrelevant factors. To expand the scope of the application, we adopt unsupervised domain adaptation to transfer the learned disentangled representations to other working environments.
  arXiv  Detail & Related papers  (2023-08-06T05:58:45Z)
• Self-Supervised Training with Autoencoders for Visual Anomaly Detection [61.62861063776813]
  We focus on a specific use case in anomaly detection where the distribution of normal samples is supported by a lower-dimensional manifold. We adapt a self-supervised learning regime that exploits discriminative information during training but focuses on the submanifold of normal examples. We achieve a new state-of-the-art result on the MVTec AD dataset -- a challenging benchmark for visual anomaly detection in the manufacturing domain.
  arXiv  Detail & Related papers  (2022-06-23T14:16:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.