Why Low-Precision Transformer Training Fails: An Analysis on Flash Attention

• URL: http://arxiv.org/abs/2510.04212v2
• Date: Fri, 10 Oct 2025 11:14:31 GMT
• Title: Why Low-Precision Transformer Training Fails: An Analysis on Flash Attention
• Abstract summary: This paper provides the first mechanistic explanation for a long-standing and unresolved failure case where training with flash attention in low-precision settings leads to catastrophic loss explosion.<n>Our in-depth analysis reveals that the failure is not a random artifact but caused by two intertwined phenomena: the emergence of similar low-rank representations within the attention mechanism and the compounding effect of biased rounding errors inherent in low-precision arithmetic.
• Score: 37.00916406627862
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The pursuit of computational efficiency has driven the adoption of low-precision formats for training transformer models. However, this progress is often hindered by notorious training instabilities. This paper provides the first mechanistic explanation for a long-standing and unresolved failure case where training with flash attention in low-precision settings leads to catastrophic loss explosion. Our in-depth analysis reveals that the failure is not a random artifact but caused by two intertwined phenomena: the emergence of similar low-rank representations within the attention mechanism and the compounding effect of biased rounding errors inherent in low-precision arithmetic. We demonstrate how these factors create a vicious cycle of error accumulation that corrupts weight updates, ultimately derailing the training dynamics. To validate our findings, we introduce a minimal modification to the flash attention that mitigates the bias in rounding errors. This simple change stabilizes the training process, confirming our analysis and offering a practical solution to this persistent problem. Code is available at https://github.com/ucker/why-low-precision-training-fails.

Related papers

• Fine-Tuning Without Forgetting via Loss-Adaptive Learning Rates [10.06646055401747]
  Fine-tuning large language models on new data improves task performance but degrades capabilities learned during pretraining.<n>Existing methods mitigate this by modifying the fine-tuning objective to suppress high-loss tokens or sequences.<n>We introduce FINCH, a loss-adaptive learning-rate schedule that reduces forgetting by 93% on average while matching the task performance of standard fine-tuning.
  arXiv  Detail & Related papers  (2026-05-19T15:36:52Z)
• Rethinking Early Stopping: Refine, Then Calibrate [49.966899634962374]
  We present a novel variational formulation of the calibration-refinement decomposition.<n>We provide theoretical and empirical evidence that calibration and refinement errors are not minimized simultaneously during training.
  arXiv  Detail & Related papers  (2025-01-31T15:03:54Z)
• Abrupt Learning in Transformers: A Case Study on Matrix Completion [15.210510215283882]
  We formulate the low-rank matrix completion problem as a masked language modeling (MLM) task. We show that it is possible to train a BERT model to solve this task to low error. We also analyze the training dynamics of individual model components to understand the sudden drop in loss.
  arXiv  Detail & Related papers  (2024-10-29T17:08:06Z)
• SINDER: Repairing the Singular Defects of DINOv2 [61.98878352956125]
  Vision Transformer models trained on large-scale datasets often exhibit artifacts in the patch token they extract. We propose a novel fine-tuning smooth regularization that rectifies structural deficiencies using only a small dataset.
  arXiv  Detail & Related papers  (2024-07-23T20:34:23Z)
• Is Flash Attention Stable? [12.827526286642282]
  Many organizations training state-of-the-art Generative AI models have reported isolated cases of instability during training. We develop a principled approach to understanding the effects of numeric deviation, and construct proxies to put observations into context. We find that Flash Attention sees roughly an order magnitude of more numeric deviation as compared to Baseline Attention at BF16 when measured during a forward pass.
  arXiv  Detail & Related papers  (2024-05-05T03:25:25Z)
• Unmasking Bias in Diffusion Model Training [40.90066994983719]
  Denoising diffusion models have emerged as a dominant approach for image generation. They still suffer from slow convergence in training and color shift issues in sampling. In this paper, we identify that these obstacles can be largely attributed to bias and suboptimality inherent in the default training paradigm.
  arXiv  Detail & Related papers  (2023-10-12T16:04:41Z)
• Dropout Reduces Underfitting [85.61466286688385]
  In this study, we demonstrate that dropout can also mitigate underfitting when used at the start of training. We find dropout reduces the directional variance of gradients across mini-batches and helps align the mini-batch gradients with the entire dataset's gradient. Our findings lead us to a solution for improving performance in underfitting models - early dropout: dropout is applied only during the initial phases of training, and turned off afterwards.
  arXiv  Detail & Related papers  (2023-03-02T18:59:15Z)
• Input Perturbation Reduces Exposure Bias in Diffusion Models [41.483581603727444]
  We show that a long sampling chain leads to an error accumulation phenomenon, similar to the exposure bias problem in autoregressive text generation. We propose a very simple but effective training regularization, consisting in perturbing the ground truth samples to simulate the inference time prediction errors. We empirically show that, without affecting the recall and precision, the proposed input perturbation leads to a significant improvement in the sample quality.
  arXiv  Detail & Related papers  (2023-01-27T13:34:54Z)
• Calibrated Adversarial Training [8.608288231153304]
  We present the Calibrated Adversarial Training, a method that reduces the adverse effects of semantic perturbations in adversarial training. The method produces pixel-level adaptations to the perturbations based on novel calibrated robust error.
  arXiv  Detail & Related papers  (2021-10-01T19:17:28Z)
• Overfitting or Underfitting? Understand Robustness Drop in Adversarial Training [34.83228408320053]
  We propose APART, an adaptive adversarial training framework, which parameterizes perturbation generation and progressively strengthens them. APART provides comparable or even better robustness than PGD-10, with only about 1/4 of its computational cost.
  arXiv  Detail & Related papers  (2020-10-15T21:43:07Z)
• Step-Ahead Error Feedback for Distributed Training with Compressed Gradient [99.42912552638168]
  We show that a new "gradient mismatch" problem is raised by the local error feedback in centralized distributed training. We propose two novel techniques, 1) step ahead and 2) error averaging, with rigorous theoretical analysis.
  arXiv  Detail & Related papers  (2020-08-13T11:21:07Z)
• Understanding the Difficulty of Training Transformers [120.99980924577787]
  We show that unbalanced gradients are not the root cause of the instability of training. We propose Admin to stabilize the early stage's training and unleash its full potential in the late stage.
  arXiv  Detail & Related papers  (2020-04-17T13:59:07Z)
• Overfitting in adversarially robust deep learning [86.11788847990783]
  We show that overfitting to the training set does in fact harm robust performance to a very large degree in adversarially robust training. We also show that effects such as the double descent curve do still occur in adversarially trained models, yet fail to explain the observed overfitting.
  arXiv  Detail & Related papers  (2020-02-26T15:40:50Z)
• Self-Adaptive Training: beyond Empirical Risk Minimization [15.59721834388181]
  We propose a new training algorithm that dynamically corrects problematic labels by model predictions without incurring extra computational cost. Self-adaptive training significantly improves generalization over various levels of noises, and mitigates the overfitting issue in both natural and adversarial training. Experiments on CIFAR and ImageNet datasets verify the effectiveness of our approach in two applications.
  arXiv  Detail & Related papers  (2020-02-24T15:47:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.