Large-Step Training Dynamics of a Two-Factor Linear Transformer Model

• URL: http://arxiv.org/abs/2605.21292v1
• Date: Wed, 20 May 2026 15:25:13 GMT
• Title: Large-Step Training Dynamics of a Two-Factor Linear Transformer Model
• Abstract summary: We study an exactly reducible one-prompt linear-transformer training problem.<n>After normalization, the dynamics reduce to a two-factor product map.<n>We show that large constant learning rates can change the training attractor of the learned transformer.
• Score: 5.256913964154542
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Gradient-flow analyses show that simplified linear transformers can learn the in-context linear-regression algorithm, but they do not explain the finite-step behavior of gradient descent at large learning rates. Motivated by empirical work on high-learning-rate transformer instabilities and by the cubic-map phase diagram for quadratic regression, we study an exactly reducible one-prompt linear-transformer training problem. After normalization, the dynamics reduce to a two-factor product map with an effective step-size parameter \(μ\). On the balanced slice, this map recovers the known scalar cubic transition from monotone convergence to catapult convergence, periodic and chaotic bounded nonconvergence, and divergence. We then analyze the full two-dimensional system and show that, for \(0<μ<2\), it has an explicit invariant Chebyshev ellipse separating forward-invariant regions; this ellipse carries off-balanced chaotic dynamics but is transversely repelling, while balanced scalar attractors can be transversely attracting. These results show that large constant learning rates can change the training attractor of the learned transformer rather than merely accelerating convergence: beyond sharp stability thresholds, finite-step training may settle into cycles, bounded chaos, or divergence instead of a single in-context linear-regression solution. We also discuss the consequences for mini-batch gradient descent based training methods.

Related papers

• Implicit Bias of SGD in Multivariate ReLU Networks: Effective Width Collapse [16.78534503577613]
  We show that the input weights and biases align along finitely many directions, leading to an effective width collapse.<n>We establish a non-redundancy property of the learned representation by proving that each learned direction induces a unique ternary activation pattern on the training data.
  arXiv  Detail & Related papers  (2026-07-03T22:03:13Z)
• Representational Curvature Modulates Behavioral Uncertainty in Large Language Models [7.361948338002342]
  temporal straightening offers an account of how the next-token prediction objective shapes representations.<n>In large language models (LLMs), temporal straightening offers an account of how the next-token prediction objective shapes representations.<n>We provide such a link by relating contextual curvature-a geometric measure of how sharply the representational trajectory bends over recent context-to next-token entropy.
  arXiv  Detail & Related papers  (2026-04-27T03:00:47Z)
• On Disentangled Training for Nonlinear Transform in Learned Image Compression [59.66885464492666]
  Learned image compression (LIC) has demonstrated superior rate-distortion (R-D) performance compared to traditional codecs.<n>Existing LIC methods overlook the slow convergence caused by compacting energy in learning nonlinear transforms.<n>We propose a linear auxiliary transform (AuxT) to disentangle energy compaction in training nonlinear transforms.
  arXiv  Detail & Related papers  (2025-01-23T15:32:06Z)
• Training Dynamics of Transformers to Recognize Word Co-occurrence via Gradient Flow Analysis [97.54180451650122]
  We study the dynamics of training a shallow transformer on a task of recognizing co-occurrence of two designated words. We analyze the gradient flow dynamics of simultaneously training three attention matrices and a linear layer. We prove a novel property of the gradient flow, termed textitautomatic balancing of gradients, which enables the loss values of different samples to decrease almost at the same rate and further facilitates the proof of near minimum training loss.
  arXiv  Detail & Related papers  (2024-10-12T17:50:58Z)
• Can Looped Transformers Learn to Implement Multi-step Gradient Descent for In-context Learning? [69.4145579827826]
  We show a fast flow on the regression loss despite the gradient non-ity algorithms for our convergence landscape. This is the first theoretical analysis for multi-layer Transformer in this setting.
  arXiv  Detail & Related papers  (2024-10-10T18:29:05Z)
• Adaptive Federated Learning Over the Air [108.62635460744109]
  We propose a federated version of adaptive gradient methods, particularly AdaGrad and Adam, within the framework of over-the-air model training. Our analysis shows that the AdaGrad-based training algorithm converges to a stationary point at the rate of $mathcalO( ln(T) / T 1 - frac1alpha ).
  arXiv  Detail & Related papers  (2024-03-11T09:10:37Z)
• Geometric Dynamics of Signal Propagation Predict Trainability of Transformers [22.25628914395565]
  We investigate forward signal propagation and gradient back propagation in deep, randomly transformers. Our approach treats the evolution of $n tokens as they propagate through the transformer layers. We show through experiments that, remarkably, the final test loss at the end of training is well predicted just by these two exponents.
  arXiv  Detail & Related papers  (2024-03-05T01:30:34Z)
• Beyond the Edge of Stability via Two-step Gradient Updates [49.03389279816152]
  Gradient Descent (GD) is a powerful workhorse of modern machine learning. GD's ability to find local minimisers is only guaranteed for losses with Lipschitz gradients. This work focuses on simple, yet representative, learning problems via analysis of two-step gradient updates.
  arXiv  Detail & Related papers  (2022-06-08T21:32:50Z)
• On regularization of gradient descent, layer imbalance and flat minima [9.08659783613403]
  We analyze the training dynamics for deep linear networks using a new metric - imbalance - which defines the flatness of a solution. We demonstrate that different regularization methods, such as weight decay or noise data augmentation, behave in a similar way.
  arXiv  Detail & Related papers  (2020-07-18T00:09:14Z)
• Path Sample-Analytic Gradient Estimators for Stochastic Binary Networks [78.76880041670904]
  In neural networks with binary activations and or binary weights the training by gradient descent is complicated. We propose a new method for this estimation problem combining sampling and analytic approximation steps. We experimentally show higher accuracy in gradient estimation and demonstrate a more stable and better performing training in deep convolutional models.
  arXiv  Detail & Related papers  (2020-06-04T21:51:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.