What Happens During the Loss Plateau? Understanding Abrupt Learning in Transformers

• URL: http://arxiv.org/abs/2506.13688v1
• Date: Mon, 16 Jun 2025 16:51:18 GMT
• Title: What Happens During the Loss Plateau? Understanding Abrupt Learning in Transformers
• Abstract summary: This work investigates the underlying mechanisms for such dynamics, primarily in shallow Transformers.<n>We reveal that during the plateau, the model often develops an interpretable partial solution while simultaneously exhibiting a strong repetition bias in their outputs.<n>We validate that these identified phenomena-repetition bias and representation collapse-are not artifacts of toy setups but also manifest in the early pre-training stage of large language models like Pythia and OLMo.
• Score: 9.575216516290237
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Training Transformers on algorithmic tasks frequently demonstrates an intriguing abrupt learning phenomenon: an extended performance plateau followed by a sudden, sharp improvement. This work investigates the underlying mechanisms for such dynamics, primarily in shallow Transformers. We reveal that during the plateau, the model often develops an interpretable partial solution while simultaneously exhibiting a strong repetition bias in their outputs. This output degeneracy is accompanied by internal representation collapse, where hidden states across different tokens become nearly parallel. We further identify the slow learning of optimal attention maps as a key bottleneck. Hidden progress in attention configuration during the plateau precedes the eventual rapid convergence, and directly intervening on attention significantly alters plateau duration and the severity of repetition bias and representational collapse. We validate that these identified phenomena-repetition bias and representation collapse-are not artifacts of toy setups but also manifest in the early pre-training stage of large language models like Pythia and OLMo.

Related papers

• Emergent Capabilities Arise Randomly from Learning Sparse Attention Patterns [48.062049009807446]
  downstream capabilities such as in-context learning are known to emerge abruptly past a certain model scale.<n>We demonstrate that the emergence of capabilities such as pattern completion and indirect object identification corresponds to the abrupt learning of task-relevant attention patterns.<n>We train transformer models on synthetic linear map and cellular automata datasets, and we show that the difficulty of learning attention patterns depends on context length and pattern sparsity.
  arXiv  Detail & Related papers  (2026-06-23T17:51:10Z)
• Phase Transitions in Attention: A Bayesian Theory of Copy Head Emergence [6.025438902954768]
  We show how the copy subcircuit in the first layer of an induction head is learned by analyzing a single-layer softmax attention network trained on a copy task.<n>Our work provides a first-principles theoretical account of the abrupt emergence of the copy subcircuit, reminiscent of the one observed in training large language models.
  arXiv  Detail & Related papers  (2026-06-10T13:26:56Z)
• Two Speeds of Learning: A Representation-Readout Decomposition of Grokking and Double Descent [7.1799939083022695]
  Two well-documented phenomena complicate this picture: in grokking, train loss falls rapidly while test performance improves abruptly only after a long delay.<n>We analyze two competing processes that underlie learning dynamics: representation learning in the encoder and readout calibration in the final classifier.<n>Applying the representation-readout decomposition to grokking across a wide range of tasks and architectures, we find that the readout is train-biased before grokking onset.
  arXiv  Detail & Related papers  (2026-05-26T14:26:55Z)
• A Progressive Training Strategy for Vision-Language Models to Counteract Spatio-Temporal Hallucinations in Embodied Reasoning [49.61652671596548]
  "Multi-image hallucination reasoning", where a massive performance drop between forward and temporal queries reveals a dependence on superficial shortcuts instead of genuine understanding.<n>To mitigate this, we develop a new dataset: Chain-of-temporal building on detailed reasoning into Chain steps and definitive judgments.<n>Our experiments demonstrate that this approach not only improves accuracy but also the forward-backward performance gap from over 70% to only 6.53%.
  arXiv  Detail & Related papers  (2026-04-12T07:48:44Z)
• The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks [32.60957674853853]
  We study two recurring phenomena in Transformer language models.<n>Massive activations, in which a small number of tokens exhibit extreme outliers in a few channels, and attention sinks, in which certain tokens attract disproportionate attention mass regardless of semantic relevance.
  arXiv  Detail & Related papers  (2026-03-05T18:59:04Z)
• How Attention Sinks Emerge in Large Language Models: An Interpretability Perspective [67.08306259095778]
  We identify a simple mechanism that enables the model to recognize token at position zero and induce an attention sink within two transformer blocks.<n>By analyzing training traces from a 30B A3B MoE model trained from scratch, we find that this mechanism emerges early in training and becomes increasingly concentrated in the first two layers.
  arXiv  Detail & Related papers  (2026-02-04T11:10:34Z)
• Mechanistic Analysis of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning [0.0]
  Large language models exhibit remarkable performance across diverse tasks through pre-training and fine-tuning paradigms.<n>Continual fine-tuning on sequential tasks induces catastrophic forgetting, where newly acquired knowledge interferes with previously learned capabilities.<n>We identify three primary mechanisms driving forgetting: gradient interference in attention weights, representational drift in intermediate layers, and loss landscape flattening.
  arXiv  Detail & Related papers  (2026-01-26T17:15:10Z)
• Understanding Transformers for Time Series: Rank Structure, Flow-of-ranks, and Compressibility [90.894232610821]
  We analyze Transformers through the lens of rank structure.<n>We show that time-series embeddings exhibit sharply decaying singular value spectra.<n>We prove that the associated $Q/K/V$ projections admit accurate low-rank approximations.
  arXiv  Detail & Related papers  (2025-10-02T23:56:17Z)
• The emergence of sparse attention: impact of data distribution and benefits of repetition [14.652502263025882]
  We study the emergence over training of sparse attention, a critical and frequently observed attention pattern in Transformers.<n>By combining theoretical analysis of a toy model with empirical observations on small Transformers trained on a linear regression variant, we uncover the mechanics sparse attention emergence.<n>Our findings provide a simple, theoretically grounded framework for understanding how data distributions and model design influence the learning dynamics behind one form of emergence.
  arXiv  Detail & Related papers  (2025-05-23T13:14:02Z)
• New Evidence of the Two-Phase Learning Dynamics of Neural Networks [59.55028392232715]
  We introduce an interval-wise perspective that compares network states across a time window.<n>We show that the response of the network to a perturbation exhibits a transition from chaotic to stable.<n>We also find that after this transition point the model's functional trajectory is confined to a narrow cone-shaped subset.
  arXiv  Detail & Related papers  (2025-05-20T04:03:52Z)
• On the Emergence of Position Bias in Transformers [59.87743433861665]
  This paper presents a graph-theoretic framework for analyzing position biases in multilayer positions.<n>Our framework offers a principled foundation for understanding positional interplay in transformers.
  arXiv  Detail & Related papers  (2025-02-04T02:53:07Z)
• Abrupt Learning in Transformers: A Case Study on Matrix Completion [15.210510215283882]
  We formulate the low-rank matrix completion problem as a masked language modeling (MLM) task. We show that it is possible to train a BERT model to solve this task to low error. We also analyze the training dynamics of individual model components to understand the sudden drop in loss.
  arXiv  Detail & Related papers  (2024-10-29T17:08:06Z)
• Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMs [77.66717051042032]
  Practitioners have consistently observed three puzzling phenomena in transformer-based large language models. These phenomena are characterized by certain so-called "sink tokens" receiving disproportionately high attention weights. We elucidate the mechanisms behind extreme-token phenomena.
  arXiv  Detail & Related papers  (2024-10-17T17:54:06Z)
• Uncovering Layer-Dependent Activation Sparsity Patterns in ReLU Transformers [2.1572258716881905]
  We explore how token-level sparsity evolves over the course of training, and how it connects to broader sparsity patterns. In particular, we demonstrate that the first and last layer of the network have distinctive and in many ways inverted relationships to sparsity. We additionally explore the phenomenon of ReLU dimensions "turning off", and show evidence suggesting that "neuron death" is being driven by the dynamics of training.
  arXiv  Detail & Related papers  (2024-07-10T17:10:10Z)
• Diagnosing Catastrophe: Large parts of accuracy loss in continual learning can be accounted for by readout misalignment [0.0]
  Training artificial neural networks on changing data distributions leads to a rapid decrease in performance on old tasks. We investigate the representational changes that underlie this performance decrease and identify three distinct processes that together account for the phenomenon.
  arXiv  Detail & Related papers  (2023-10-09T11:57:46Z)
• Stabilizing Transformer Training by Preventing Attention Entropy Collapse [56.45313891694746]
  We investigate the training dynamics of Transformers by examining the evolution of the attention layers. We show that $sigma$Reparam successfully prevents entropy collapse in the attention layers, promoting more stable training. We conduct experiments with $sigma$Reparam on image classification, image self-supervised learning, machine translation, speech recognition, and language modeling tasks.
  arXiv  Detail & Related papers  (2023-03-11T03:30:47Z)
• Exploring Transferable and Robust Adversarial Perturbation Generation from the Perspective of Network Hierarchy [52.153866313879924]
  The transferability and robustness of adversarial examples are two practical yet important properties for black-box adversarial attacks. We propose a transferable and robust adversarial generation (TRAP) method. Our TRAP achieves impressive transferability and high robustness against certain interferences.
  arXiv  Detail & Related papers  (2021-08-16T11:52:41Z)
• Dissecting Lottery Ticket Transformers: Structural and Behavioral Study of Sparse Neural Machine Translation [0.0]
  Recent work on the lottery ticket hypothesis has produced highly sparse Transformers for NMT while maintaining BLEU. By probing Transformers with more and more low-magnitude weights pruned away, we find that complex semantic information is first to be degraded. Analysis of internal activations reveals that higher layers diverge most over the course of pruning, gradually becoming less complex than their dense counterparts.
  arXiv  Detail & Related papers  (2020-09-17T02:08:45Z)
• Extreme Memorization via Scale of Initialization [72.78162454173803]
  We construct an experimental setup in which changing the scale of initialization strongly impacts the implicit regularization induced by SGD. We find that the extent and manner in which generalization ability is affected depends on the activation and loss function used. In the case of the homogeneous ReLU activation, we show that this behavior can be attributed to the loss function.
  arXiv  Detail & Related papers  (2020-08-31T04:53:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.