Spectral Alignment as Predictor of Loss Explosion in Neural Network Training

• URL: http://arxiv.org/abs/2510.04202v1
• Date: Sun, 05 Oct 2025 13:35:34 GMT
• Title: Spectral Alignment as Predictor of Loss Explosion in Neural Network Training
• Abstract summary: Loss explosions in training deep neural networks can nullify multi-million dollar training runs.<n>We introduce Spectral Alignment, a novel metric that monitors the distributional alignment between layer inputs and the principal singular vectors of weight matrices.<n> Empirical results on language models demonstrate that monitoring the SA distribution provides a significantly earlier and clearer warning of loss explosions than traditional scalar metrics.
• Score: 31.831122738372603
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Loss explosions in training deep neural networks can nullify multi-million dollar training runs. Conventional monitoring metrics like weight and gradient norms are often lagging and ambiguous predictors, as their values vary dramatically across different models and even between layers of the same model, making it difficult to establish a unified standard for detecting impending failure. We introduce Spectral Alignment (SA), a novel, theoretically-grounded metric that monitors the distributional alignment between layer inputs and the principal singular vectors of weight matrices. We show that a collapse in the sign diversity of this alignment is a powerful early predictor of representational collapse and training divergence. Empirical results on language models demonstrate that monitoring the SA distribution provides a significantly earlier and clearer warning of loss explosions than traditional scalar metrics. SA's low computational overhead makes it a practical tool for safeguarding model training.

Related papers

• Into the ORBIT for Time Series: Training Regimes for Foundation Models [16.359864017246494]
  ORBIT is a training paradigm that makes pre-training distributions explicit and controllable.<n>We introduce Rank-Guided Cross-Depth Alignment, a training objective that uses late-layer representations as stop-gradient teachers.<n> Evaluations on GIFT-Eval and fev-bench demonstrate strong zero-shot forecasting performance across diverse domains and frequencies.
  arXiv  Detail & Related papers  (2026-08-13T14:00:39Z)
• Mechanism-Driven Monitors for Preemptive Detection of LLM Training Instability [20.49786007379445]
  We study mechanism-driven detection of training instability by deriving internal monitors from the functional role of each critical module.<n>Our fault-injection experiments on low-precision attention, large learning-rate, and combined faults show that these signals provide distinct signatures for different failures.
  arXiv  Detail & Related papers  (2026-06-26T14:18:22Z)
• SphereVAD: Training-Free Video Anomaly Detection via Geodesic Inference on the Unit Hypersphere [90.85001795656633]
  Video anomaly detection (VAD) aims to automatically identify events that deviate from normal patterns in untrimmed surveillance videos.<n>We propose SphereVAD, a fully training-free, zero-shot VAD framework that recasts anomaly discrimination as von Mises-Fisher (vMF) likelihood-ratio geodesic inference on the unit hypersphere.
  arXiv  Detail & Related papers  (2026-05-08T16:57:38Z)
• LogitDynamics: Reliable ViT Error Detection from Layerwise Logit Trajectories [1.4610038284393168]
  We study error prediction in Vision Transformers (ViTs)<n>Motivated by internal-signal hallucination detection in large language models, we investigate whether similar depth-wise signals exist inViTs.<n>By attaching lightweight linear heads to intermediate layers, we extract features from the last L layers that capture both the logits of the predicted class and its top-K competitors.<n>A linear probe trained on these features predicts the error indicator.
  arXiv  Detail & Related papers  (2026-04-12T13:43:40Z)
• Amortized Predictability-aware Training Framework for Time Series Forecasting and Classification [10.816479922364097]
  We propose a general Amortized Predictability-aware Training Framework (APTF) for both time series forecasting (TSF) and time series classification (TSC)<n>APTF introduces two key designs that enable the model to focus on high-predictability samples while still learning appropriately from low-predictability ones.
  arXiv  Detail & Related papers  (2026-02-18T06:59:05Z)
• Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks [59.552873049024775]
  We show that compute-optimally trained models exhibit a remarkably precise universality.<n>With learning rate decay, the collapse becomes so tight that differences in the normalized curves across models fall below the noise floor.<n>We explain these phenomena by connecting collapse to the power-law structure in typical neural scaling laws.
  arXiv  Detail & Related papers  (2025-07-02T20:03:34Z)
• SINDER: Repairing the Singular Defects of DINOv2 [61.98878352956125]
  Vision Transformer models trained on large-scale datasets often exhibit artifacts in the patch token they extract. We propose a novel fine-tuning smooth regularization that rectifies structural deficiencies using only a small dataset.
  arXiv  Detail & Related papers  (2024-07-23T20:34:23Z)
• QGait: Toward Accurate Quantization for Gait Recognition with Binarized Input [17.017127559393398]
  We propose a differentiable soft quantizer, which better simulates the gradient of the round function during backpropagation. This enables the network to learn from subtle input perturbations. We further refine the training strategy to ensure convergence while simulating quantization errors.
  arXiv  Detail & Related papers  (2024-05-22T17:34:18Z)
• Unsupervised Continual Anomaly Detection with Contrastively-learned Prompt [80.43623986759691]
  We introduce a novel Unsupervised Continual Anomaly Detection framework called UCAD. The framework equips the UAD with continual learning capability through contrastively-learned prompts. We conduct comprehensive experiments and set the benchmark on unsupervised continual anomaly detection and segmentation.
  arXiv  Detail & Related papers  (2024-01-02T03:37:11Z)
• GIT: Detecting Uncertainty, Out-Of-Distribution and Adversarial Samples using Gradients and Invariance Transformations [77.34726150561087]
  We propose a holistic approach for the detection of generalization errors in deep neural networks. GIT combines the usage of gradient information and invariance transformations. Our experiments demonstrate the superior performance of GIT compared to the state-of-the-art on a variety of network architectures.
  arXiv  Detail & Related papers  (2023-07-05T22:04:38Z)
• Regularized Vector Quantization for Tokenized Image Synthesis [126.96880843754066]
  Quantizing images into discrete representations has been a fundamental problem in unified generative modeling. deterministic quantization suffers from severe codebook collapse and misalignment with inference stage while quantization suffers from low codebook utilization and reconstruction objective. This paper presents a regularized vector quantization framework that allows to mitigate perturbed above issues effectively by applying regularization from two perspectives.
  arXiv  Detail & Related papers  (2023-03-11T15:20:54Z)
• Lossy Compression for Robust Unsupervised Time-Series Anomaly Detection [4.873362301533825]
  We propose a Lossy Causal Temporal Convolutional Neural Network Autoencoder for anomaly detection. Our framework uses a rate-distortion loss and an entropy bottleneck to learn a compressed latent representation for the task.
  arXiv  Detail & Related papers  (2022-12-05T14:29:16Z)
• Self-Supervised Training with Autoencoders for Visual Anomaly Detection [61.62861063776813]
  We focus on a specific use case in anomaly detection where the distribution of normal samples is supported by a lower-dimensional manifold. We adapt a self-supervised learning regime that exploits discriminative information during training but focuses on the submanifold of normal examples. We achieve a new state-of-the-art result on the MVTec AD dataset -- a challenging benchmark for visual anomaly detection in the manufacturing domain.
  arXiv  Detail & Related papers  (2022-06-23T14:16:30Z)
• A heteroencoder architecture for prediction of failure locations in porous metals using variational inference [1.2722697496405462]
  We employ an encoder-decoder convolutional neural network to predict the failure locations of porous metal tension specimens. The objective of predicting failure locations presents an extreme case of class imbalance since most of the material in the specimens do not fail. We demonstrate that the resulting predicted variances are effective in ranking the locations that are most likely to fail in any given specimen.
  arXiv  Detail & Related papers  (2022-01-31T20:26:53Z)
• Dense Out-of-Distribution Detection by Robust Learning on Synthetic Negative Data [1.7474352892977458]
  We show how to detect out-of-distribution anomalies in road-driving scenes and remote sensing imagery. We leverage a jointly trained normalizing flow due to coverage-oriented learning objective and the capability to generate samples at different resolutions. The resulting models set the new state of the art on benchmarks for out-of-distribution detection in road-driving scenes and remote sensing imagery.
  arXiv  Detail & Related papers  (2021-12-23T20:35:10Z)
• Shaping Deep Feature Space towards Gaussian Mixture for Visual Classification [74.48695037007306]
  We propose a Gaussian mixture (GM) loss function for deep neural networks for visual classification. With a classification margin and a likelihood regularization, the GM loss facilitates both high classification performance and accurate modeling of the feature distribution. The proposed model can be implemented easily and efficiently without using extra trainable parameters.
  arXiv  Detail & Related papers  (2020-11-18T03:32:27Z)
• Asymptotic Behavior of Adversarial Training in Binary Classification [41.7567932118769]
  Adversarial training is considered to be the state-of-the-art method for defense against adversarial attacks. Despite being successful in practice, several problems in understanding performance of adversarial training remain open. We derive precise theoretical predictions for the minimization of adversarial training in binary classification.
  arXiv  Detail & Related papers  (2020-10-26T01:44:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.