Residual Koopman Spectral Profiling for Predicting and Preventing Transformer Training Instability

• URL: http://arxiv.org/abs/2602.22988v1
• Date: Thu, 26 Feb 2026 13:33:25 GMT
• Title: Residual Koopman Spectral Profiling for Predicting and Preventing Transformer Training Instability
• Abstract summary: Training divergence in transformers wastes compute, yet practitioners discover instability only after expensive runs begin.<n>Our study of Residual Koopman Spectral Profiling (RKSP) provides such an estimate.<n>Our central diagnostic, the near-unit spectral mass, quantifies the fraction of modes concentrated near the unit circle, which captures instability risk.
• Score: 40.24077350481118
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Training divergence in transformers wastes compute, yet practitioners discover instability only after expensive runs begin. They therefore need an expected probability of failure for a transformer before training starts. Our study of Residual Koopman Spectral Profiling (RKSP) provides such an estimate. From a single forward pass at initialization, RKSP extracts Koopman spectral features by applying whitened dynamic mode decomposition to layer-wise residual snapshots. Our central diagnostic, the near-unit spectral mass, quantifies the fraction of modes concentrated near the unit circle, which captures instability risk. For predicting divergence across extensive configurations, this estimator achieves an AUROC of 0.995, outperforming the best gradient baseline. We further make this diagnostic actionable through Koopman Spectral Shaping (KSS), which reshapes spectra during training. We empirically validate that our method works in practice: RKSP predicts divergence at initialization, and when RKSP flags high risk, turning on KSS successfully prevents divergence. In the challenging high learning rate regime without normalization layers, KSS reduces the divergence rate from 66.7% to 12.5% and enables learning rates that are 50% to 150% higher. These findings generalize to WikiText-103 language modeling, vision transformers on CIFAR-10, and pretrained language models, including GPT-2 and LLaMA-2 up to 7B, as well as emerging architectures such as MoE, Mamba-style SSMs, and KAN.

Related papers

• Propensity Straight-Through Gradients for Discrete Stochastic Systems [0.0]
  We exploit the affine state update to obtain the exact one-step conditional-mean sensitivity.<n>We show that one-step sensitivities composed across events can depart from the exact multistep sensitivity.
  arXiv  Detail & Related papers  (2026-08-26T10:59:25Z)
• SPARC: Single-Pass Scaling for Motion Forecasting with Conformal Bayesian Last Layers [1.4452289368758382]
  Single-Pass Adaptive Risk is a structured, calibrated, and efficient uncertainty estimate for human motion forecasters.<n>A deterministic backbone predicts the future mean, and a conjugate Bayesian last layer converts time-domain feature leverage into an analytic horizon-wise scale.<n>It ranks first on NLL and on the combined MPJPE+NLL criterion while retaining competitive point accuracy and efficient calibrated tubes.
  arXiv  Detail & Related papers  (2026-08-21T07:20:22Z)
• Learning to Refine: Spectral-Decoupled Iterative Refinement Framework for Precipitation Nowcasting [21.300422393514804]
  We propose a deterministic framework that reformulates nowcasting as progressive frequency-decoupled refinement.<n>SDIR first extracts a stable low-frequency synoptic skeleton, then iteratively refines high-frequency textures under physical constraints.<n> Experiments on three benchmarks show SDIR significantly outperforms SOTA methods in spatial accuracy.
  arXiv  Detail & Related papers  (2026-06-01T07:34:30Z)
• Clipping Bottleneck: Stabilizing RLVR via Stochastic Recovery of Near-Boundary Signals [83.0127582612634]
  Near-boundary Rescue (NSR) is a minimal, plug-and-play modification that retains out-of-bound tokens to recover lost signals.<n>NSR substantially improves training stability and delivers consistent gains over strong baselines such as DAPO and GSPO.
  arXiv  Detail & Related papers  (2026-05-21T16:45:31Z)
• Low-Rank Compression of Pretrained Models via Randomized Subspace Iteration [2.1016374925364616]
  Low-rank decomposition based on the singular value decomposition (SVD) provides a principled approach for model reduction.<n> Randomized alternatives such as randomized SVD (RSVD) improve efficiency, yet they suffer from poor approximation quality when the singular value spectrum decays slowly.<n>We propose randomized subspace (RSI) as a more effective alternative.
  arXiv  Detail & Related papers  (2026-04-03T02:47:03Z)
• BadCLIP++: Stealthy and Persistent Backdoors in Multimodal Contrastive Learning [73.46118996284888]
  Research on backdoor attacks against multimodal contrastive learning models faces two key challenges: stealthiness and persistence.<n>We propose BadCLIP++, a unified framework that tackles both challenges.<n>For stealthiness, we introduce a semantic-fusion QR micro-trigger that embeds imperceptible patterns near task-relevant regions.<n>For persistence, we stabilize trigger embeddings via radius shrinkage and centroid alignment.
  arXiv  Detail & Related papers  (2026-02-19T08:31:16Z)
• Activation-Space Uncertainty Quantification for Pretrained Networks [2.001149416674759]
  We introduce Gaussian Process Activations (GAPA), a post-hoc method that shifts Bayesian modeling from weights to activations.<n>GAPA replaces standard nonlinearities with activations whose posterior mean exactly matches the original activation, preserving the backbone's point predictions by construction.<n>To scale to modern architectures, we use a sparse variational inducing-point approximation over cached training activations, combined with local k-nearest-neighbor conditioning.
  arXiv  Detail & Related papers  (2026-02-16T17:17:08Z)
• Spectral Gating Networks [65.9496901693099]
  We introduce Spectral Gating Networks (SGN) to introduce frequency-rich expressivity in feed-forward networks.<n>SGN augments a standard activation pathway with a compact spectral pathway and learnable gates that allow the model to start from a stable base behavior.<n>It consistently improves accuracy-efficiency trade-offs under comparable computational budgets.
  arXiv  Detail & Related papers  (2026-02-07T20:00:49Z)
• Denoising and Baseline Correction of Low-Scan FTIR Spectra: A Benchmark of Deep Learning Models Against Traditional Signal Processing [0.0]
  We propose a physics-informed cascade Unet that separates denoising and baseline correction tasks.<n>This architecture forces the network to separate random noise from chemical signals using an embedded SNIP layer.<n>We benchmarked this approach against a standard single Unet and a traditional Savitzky-Golay/SNIP workflow.
  arXiv  Detail & Related papers  (2026-01-28T15:19:02Z)
• Controllable Probabilistic Forecasting with Stochastic Decomposition Layers [1.3995263206621]
  We introduce Decomposition Layers (SDL) for converting deterministic machine learning weather models into ensemble systems.<n>SDL applies learned perturbations at three decoder scales through latent-driven modulation, per-pixel noise, and channel scaling.<n>When applied to WXFormer via transfer learning, SDL requires less than 2% of the computational cost needed to train the baseline model.
  arXiv  Detail & Related papers  (2025-12-21T17:10:00Z)
• Latent Sculpting for Zero-Shot Generalization: A Manifold Learning Approach to Out-of-Distribution Anomaly Detection [2.8547732086436306]
  A fundamental limitation of supervised deep learning is "Generalization Collapse"<n>We propose Latent Sculpting, a hierarchical two-stage representation learning framework.<n>We report an 88.89% detection rate on "Infiltration" scenarios.
  arXiv  Detail & Related papers  (2025-12-19T11:37:02Z)
• deep-REMAP: Probabilistic Parameterization of Stellar Spectra Using Regularized Multi-Task Learning [0.0]
  deep-REMAP is a novel deep learning framework that utilizes a regularized, multi-task approach to predict stellar atmospheric parameters from observed spectra.<n>We train a deep convolutional neural network on the PHOENIX synthetic spectral library and use transfer learning to fine-tune the model on a small subset of observed FGK dwarf spectra.<n>Deep-REMAP accurately recovers the effective temperature ($T_rmeff$), surface gravity ($log rmg$), and metallicity ([Fe/H]), achieving a precision of, for instance, approximately 75 K in $T
  arXiv  Detail & Related papers  (2025-10-10T13:20:06Z)
• CLIPure: Purification in Latent Space via CLIP for Adversarially Robust Zero-Shot Classification [65.46685389276443]
  We ground our work on CLIP, a vision-language pre-trained encoder model that can perform zero-shot classification by matching an image with text prompts.<n>We then formulate purification risk as the KL divergence between the joint distributions purification process.<n>We propose two variants for our CLIPure approach: CLI-Diff which models the likelihood of images' latent vectors, and CLIPure-Cos which models the likelihood with the cosine similarity between the embeddings of an image and a photo of a.''
  arXiv  Detail & Related papers  (2025-02-25T13:09:34Z)
• Coarse-to-Fine Sparse Transformer for Hyperspectral Image Reconstruction [138.04956118993934]
  We propose a novel Transformer-based method, coarse-to-fine sparse Transformer (CST) CST embedding HSI sparsity into deep learning for HSI reconstruction. In particular, CST uses our proposed spectra-aware screening mechanism (SASM) for coarse patch selecting. Then the selected patches are fed into our customized spectra-aggregation hashing multi-head self-attention (SAH-MSA) for fine pixel clustering and self-similarity capturing.
  arXiv  Detail & Related papers  (2022-03-09T16:17:47Z)
• Semi-supervised Contrastive Learning with Similarity Co-calibration [72.38187308270135]
  We propose a novel training strategy, termed as Semi-supervised Contrastive Learning (SsCL) SsCL combines the well-known contrastive loss in self-supervised learning with the cross entropy loss in semi-supervised learning. We show that SsCL produces more discriminative representation and is beneficial to few shot learning.
  arXiv  Detail & Related papers  (2021-05-16T09:13:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.