Non-normal spectral signatures of instability in neural network training dynamics

• URL: http://arxiv.org/abs/2605.23476v1
• Date: Fri, 22 May 2026 10:36:48 GMT
• Title: Non-normal spectral signatures of instability in neural network training dynamics
• Abstract summary: We show that the linearized update operators for practically useds are generically non-normal.<n>Applying non-normal stability theory to these operators, we derive a conservative pseudospectral precursor bound in which (V) serves as an early-warning indicator of transient amplification.
• Score: 0.5307735857768683
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Training instabilities in deep networks - loss spikes, oscillatory convergence, and gradient pathologies - are empirically prevalent but lack a rigorous operator-theoretic explanation. We show that the linearized update operators for practically used optimizers are generically non-normal: for Adam, non-normality is controlled by the commutator [H, M] between the Hessian and the diagonal adaptive preconditioner, while for SGD with momentum it arises from the augmented state-space structure of the update map. Applying non-normal stability theory to these operators, we derive a conservative pseudospectral precursor bound in which κ(V) serves as an early-warning indicator of transient amplification even when the spectral radius remains below one, and we establish that exceptional points of the update operator appear as the κ(V) -> \infty limiting case of this framework. Numerical experiments on two-layer networks confirm that the spectral radius ρ(J) provides no separation between stable and unstable training phases while κ(V) separates them by approximately one order of magnitude, complementing the classical sharpness criterion with a continuous severity measure of non-normal amplification. These results establish non-Hermitian operator theory as a useful and underexplored framework for neural network optimization stability, offering a diagnostic language and proof-of-concept benchmark for understanding adaptive optimization stability.

Related papers

• Geometry-aware Incremental Neural Operator for Long-Horizon PDE prediction [17.3214708475049]
  We propose a geometry-aware incremental neural operator (GeoIncNO) for stable long-horizon PDE prediction.<n>GeoIncNO predicts latent increments for residual advancement and uses lightweight low-rank projectors to regulate channel coupling within active frequency bands.<n>Experiments on six PDE benchmarks, covering 1D, 2D, and 3D dynamical systems, show that GeoIncNO consistently achieves strong prediction accuracy, improved rollout stability, and better spectral fidelity compared with competitive neural-operator baselines.
  arXiv  Detail & Related papers  (2026-07-31T07:46:35Z)
• Structure-Aware Epistemic Uncertainty Quantification for Neural Operator PDE Surrogates [10.39662017167378]
  Uncertainty quantification (UQ) must be both computationally efficient and spatially faithful.<n>We propose a structure-aware UQ scheme that exploits the modular anatomy common to modern NOs.<n> Experiments on challenging PDE benchmarks demonstrate that the proposed structure-aware design yields more reliable coverage, tighter bands, and improved residual-uncertainty alignment.
  arXiv  Detail & Related papers  (2026-02-24T10:17:24Z)
• A Unified Matrix-Spectral Framework for Stability and Interpretability in Deep Learning [0.0]
  We develop a unified framework for analyzing stability and interpretability in deep neural networks.<n>We introduce a Global Matrix Stability Index that aggregates spectral information from Jacobians, parameter gradients, Neural Tangent Kernel operators, and loss Hessians into a single stability scale.
  arXiv  Detail & Related papers  (2026-02-01T10:18:37Z)
• Avoiding Premature Collapse: Adaptive Annealing for Entropy-Regularized Structural Inference [1.7523718031184992]
  We identify a fundamental mechanism for this failure: textbfPremature Mode Collapse.<n>We propose textbfEfficient Piecewise Hybrid Adaptive Stability Control (EPH-ASC), an adaptive scheduling algorithm that monitors the stability of the inference process.
  arXiv  Detail & Related papers  (2026-01-30T14:47:18Z)
• The Procrustean Bed of Time Series: The Optimization Bias of Point-wise Loss [53.542743390809356]
  This paper aims to provide a first-principles analysis of the Expectation of Optimization Bias (EOB)<n>Our analysis reveals a fundamental paradigm paradox: the more deterministic and structured the time series, the more severe the bias by point-wise loss function.<n>We present a concrete solution that simultaneously achieves both principles via DFT or DWT.
  arXiv  Detail & Related papers  (2025-12-21T06:08:22Z)
• Analysis of Fourier Neural Operators via Effective Field Theory [11.824913874212802]
  We present a systematic effective field theory analysis of FNOs in an infinite dimensional function space.<n>We show that nonlinear activations inevitably couple frequency inputs to high frequency modes that are otherwise discarded by spectral truncation.<n>Our results quantify how nonlinearity enables neural operators to capture non-trivial features and explain why scale invariant activations and residual connections enhance feature learning in FNOs.
  arXiv  Detail & Related papers  (2025-07-29T14:10:46Z)
• Stable Nonconvex-Nonconcave Training via Linear Interpolation [51.668052890249726]
  This paper presents a theoretical analysis of linearahead as a principled method for stabilizing (large-scale) neural network training. We argue that instabilities in the optimization process are often caused by the nonmonotonicity of the loss landscape and show how linear can help by leveraging the theory of nonexpansive operators.
  arXiv  Detail & Related papers  (2023-10-20T12:45:12Z)
• Benign Overfitting in Deep Neural Networks under Lazy Training [72.28294823115502]
  We show that when the data distribution is well-separated, DNNs can achieve Bayes-optimal test error for classification. Our results indicate that interpolating with smoother functions leads to better generalization.
  arXiv  Detail & Related papers  (2023-05-30T19:37:44Z)
• Stability and Generalization Analysis of Gradient Methods for Shallow Neural Networks [59.142826407441106]
  We study the generalization behavior of shallow neural networks (SNNs) by leveraging the concept of algorithmic stability. We consider gradient descent (GD) and gradient descent (SGD) to train SNNs, for both of which we develop consistent excess bounds.
  arXiv  Detail & Related papers  (2022-09-19T18:48:00Z)
• On the Double Descent of Random Features Models Trained with SGD [78.0918823643911]
  We study properties of random features (RF) regression in high dimensions optimized by gradient descent (SGD) We derive precise non-asymptotic error bounds of RF regression under both constant and adaptive step-size SGD setting. We observe the double descent phenomenon both theoretically and empirically.
  arXiv  Detail & Related papers  (2021-10-13T17:47:39Z)
• Revisiting Initialization of Neural Networks [72.24615341588846]
  We propose a rigorous estimation of the global curvature of weights across layers by approximating and controlling the norm of their Hessian matrix. Our experiments on Word2Vec and the MNIST/CIFAR image classification tasks confirm that tracking the Hessian norm is a useful diagnostic tool.
  arXiv  Detail & Related papers  (2020-04-20T18:12:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.