Taming Curvature: Architecture Warm-Up for Stable Transformer Training

• URL: http://arxiv.org/abs/2606.16768v1
• Date: Mon, 15 Jun 2026 14:16:56 GMT
• Title: Taming Curvature: Architecture Warm-Up for Stable Transformer Training
• Abstract summary: We introduce a fast online estimator of the largest (preconditioned) Hessian eigenvalue (i.e., curvature) based on a warm-started variant for power with Hessian-vector products.<n>We find that training instabilities coincide with surges in preconditioned curvature and that curvature grows with depth.<n> Experiments on large Transformers validate that our approach enables efficient curvature tracking and reduces instabilities.
• Score: 52.6838951211596
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Training billion-parameter Transformers is often brittle, with transient loss spikes and divergence that waste compute. Even though the recently developed Edge of Stability (EoS) theory provides a powerful tool to understand and control the stability of optimization methods via the (preconditioned) curvature, these curvature-controlling methods are not popular in large-scale Transformer training due to the complexity of curvature estimation. To this end, we first introduce a fast online estimator of the largest (preconditioned) Hessian eigenvalue (i.e., curvature) based on a warm-started variant for power iteration with Hessian-vector products. We show theoretically, and verify empirically, that the proposed method makes per-iteration curvature tracking feasible at billion parameter scale while being more accurate. Using this tool, we find that training instabilities coincide with surges in preconditioned curvature and that curvature grows with depth. Motivated by these observations, we propose architecture warm-up: progressively growing network depth to carefully control the preconditioned Hessian and stabilize training. Experiments on large Transformers validate that our approach enables efficient curvature tracking and reduces instabilities compared to existing state-of-the-art stabilization techniques without slowing down convergence.

Related papers

• Stable Transformer-Actor-Critic Model Predictive Control: A Contraction Analysis Approach [1.7625675658723843]
  Actor-Critic Predictive Control (MPC) effectively addresses complex, non-reaching control problems.<n>We introduce a novel Transformer-Actor-Critic MPC architecture with formal robustness architecture.
  arXiv  Detail & Related papers  (2026-06-18T13:11:21Z)
• Exponential Convergence of (Stochastic) Gradient Descent for Separable Logistic Regression [14.718691362208622]
  We show that gradient descent with a simple, non-adaptive increasing step-size schedule achieves exponential convergence for separable logistic regression under a margin condition.<n>We also establish exponential convergence of gradient descent using a lightweight adaptive step-size rule that avoids line search and specialized procedures.
  arXiv  Detail & Related papers  (2026-02-21T19:31:07Z)
• Mitigating Staleness in Asynchronous Pipeline Parallelism via Basis Rotation [6.710608163117798]
  Asynchronous pipeline parallelism maximizes hardware utilization by eliminating the pipeline bubbles inherent in synchronous execution.<n>This efficiency gain can be compromised by gradient staleness, where the immediate model updates with delayed gradients introduce noise into the optimization process.<n>We propose the use of basis rotation, demonstrating that it effectively mitigates the alignment issue and significantly accelerates convergence in asynchronous settings.
  arXiv  Detail & Related papers  (2026-02-03T13:31:51Z)
• Zero-Variance Gradients for Variational Autoencoders [32.818968022327866]
  Training deep generative models like Variational Autoencoders (VAEs) is often hindered by the need to backpropagate gradients through sampling of their latent variables.<n>In this paper, we propose a new perspective that sidesteps this problem, which we call Silent Gradients.<n>Instead of improving estimators, we leverage specific decoder architectures analytically to compute the expected ELBO, yielding a gradient with zero variance.
  arXiv  Detail & Related papers  (2025-08-05T15:54:21Z)
• Implicit factorized transformer approach to fast prediction of turbulent channel flows [6.70175842351963]
  We introduce a modified implicit factorized transformer (IFactFormer-m) model which replaces the original chained factorized attention with parallel factorized attention.<n>The IFactFormer-m model successfully performs long-term predictions for turbulent channel flow.
  arXiv  Detail & Related papers  (2024-12-25T09:05:14Z)
• Can Looped Transformers Learn to Implement Multi-step Gradient Descent for In-context Learning? [69.4145579827826]
  We show a fast flow on the regression loss despite the gradient non-ity algorithms for our convergence landscape. This is the first theoretical analysis for multi-layer Transformer in this setting.
  arXiv  Detail & Related papers  (2024-10-10T18:29:05Z)
• Adaptive Federated Learning Over the Air [108.62635460744109]
  We propose a federated version of adaptive gradient methods, particularly AdaGrad and Adam, within the framework of over-the-air model training. Our analysis shows that the AdaGrad-based training algorithm converges to a stationary point at the rate of $mathcalO( ln(T) / T 1 - frac1alpha ).
  arXiv  Detail & Related papers  (2024-03-11T09:10:37Z)
• Efficient Bound of Lipschitz Constant for Convolutional Layers by Gram Iteration [122.51142131506639]
  We introduce a precise, fast, and differentiable upper bound for the spectral norm of convolutional layers using circulant matrix theory. We show through a comprehensive set of experiments that our approach outperforms other state-of-the-art methods in terms of precision, computational cost, and scalability. It proves highly effective for the Lipschitz regularization of convolutional neural networks, with competitive results against concurrent approaches.
  arXiv  Detail & Related papers  (2023-05-25T15:32:21Z)
• Training Generative Adversarial Networks by Solving Ordinary Differential Equations [54.23691425062034]
  We study the continuous-time dynamics induced by GAN training. From this perspective, we hypothesise that instabilities in training GANs arise from the integration error. We experimentally verify that well-known ODE solvers (such as Runge-Kutta) can stabilise training.
  arXiv  Detail & Related papers  (2020-10-28T15:23:49Z)
• Extrapolation for Large-batch Training in Deep Learning [72.61259487233214]
  We show that a host of variations can be covered in a unified framework that we propose. We prove the convergence of this novel scheme and rigorously evaluate its empirical performance on ResNet, LSTM, and Transformer.
  arXiv  Detail & Related papers  (2020-06-10T08:22:41Z)
• Understanding the Difficulty of Training Transformers [120.99980924577787]
  We show that unbalanced gradients are not the root cause of the instability of training. We propose Admin to stabilize the early stage's training and unleash its full potential in the late stage.
  arXiv  Detail & Related papers  (2020-04-17T13:59:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.