How Attention Sinks Emerge in Large Language Models: An Interpretability Perspective

• URL: http://arxiv.org/abs/2603.06591v1
• Date: Wed, 04 Feb 2026 11:10:34 GMT
• Title: How Attention Sinks Emerge in Large Language Models: An Interpretability Perspective
• Abstract summary: We identify a simple mechanism that enables the model to recognize token at position zero and induce an attention sink within two transformer blocks.<n>By analyzing training traces from a 30B A3B MoE model trained from scratch, we find that this mechanism emerges early in training and becomes increasingly concentrated in the first two layers.
• Score: 67.08306259095778
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Language Models (LLMs) often allocate disproportionate attention to specific tokens, a phenomenon commonly referred to as the attention sink. While such sinks are generally considered detrimental, prior studies have identified a notable exception: the model's consistent emphasis on the first token of the input sequence. This structural bias can influence a wide range of downstream applications and warrants careful consideration. Despite its prevalence, the precise mechanisms underlying the emergence and persistence of attention sinks remain poorly understood. In this work, we trace the formation of attention sinks around the first token of the input. We identify a simple mechanism, referred to as the P0 Sink Circuit, that enables the model to recognize token at position zero and induce an attention sink within two transformer blocks, without relying on any semantic information. This mechanism serves as the basis for the attention sink on position zero. Furthermore, by analyzing training traces from a 30B A3B MoE model trained from scratch, we find that this mechanism emerges early in training and becomes increasingly concentrated in the first two layers, suggesting a possible signal for tracking pre training convergence states.

Related papers

• Disentangling Semantic Attention from Structural Bias in the Attention Manifold [61.68044165974505]
  Multimodal Large Language Models (MLLMs) exhibit disproportionate attention toward certain semantically uninformative visual tokens.<n>We introduce Saliency-guided Purification and Adaptive Redistribution (SPAR), a training-free, plug-and-play intervention.
  arXiv  Detail & Related papers  (2026-07-27T05:30:19Z)
• The Structural Origin of Attention Sink: Variance Discrepancy, Super Neurons, and Dimension Disparity [22.040532283910522]
  This work provides a textitmechanistic explanation for the phenomenon.<n>We trace its root to the value aggregation process inherent in self-attention.<n>As a proof of concept, we propose textithead-wise RMSNorm, an architectural modification that stabilizes value aggregation outputs during pre-training.
  arXiv  Detail & Related papers  (2026-05-07T17:28:55Z)
• A Mechanistic Account of Attention Sinks in GPT-2: One Circuit, Broader Implications for Mitigation [1.2744523252873352]
  Transformers commonly exhibit an attention sink: disproportionately high attention to the first position.<n>We study this behavior in GPT-2-style models with learned query biases and absolute positional embeddings.
  arXiv  Detail & Related papers  (2026-04-16T07:32:37Z)
• Attention Sink Forges Native MoE in Attention Layers: Sink-Aware Training to Address Head Collapse [11.042559854770422]
  We show that the sink in Vanilla Attention and Sink Attention naturally construct a Mixture-of-Experts (MoE) mechanism within attention layers.<n>To mitigate head collapse, we propose a sink-aware training algorithm with an auxiliary load balancing loss designed for attention layers.
  arXiv  Detail & Related papers  (2026-02-01T12:45:39Z)
• A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training [86.64715217940274]
  Outliers function jointly with normalization.<n>Outliers serve more as rescale factors rather than contributors.<n>Outliers can be absorbed into learnable parameters or mitigated via explicit gated rescaling.
  arXiv  Detail & Related papers  (2026-01-30T13:29:45Z)
• Attention Needs to Focus: A Unified Perspective on Attention Allocation [37.34801068995858]
  The Transformer architecture is a cornerstone of modern Large Language Models (LLMs)<n>Standard attention mechanism is plagued by well-documented issues: representational collapse and attention sink.<n>We present a unified perspective, arguing that both can be traced to a common root -- improper attention allocation.
  arXiv  Detail & Related papers  (2026-01-01T08:39:15Z)
• Attention Sinks: A 'Catch, Tag, Release' Mechanism for Embeddings [16.950215926321558]
  Large language models (LLMs) often concentrate their attention on a few specific tokens referred to as attention sinks.<n>Common examples include the first token, a prompt-independent sink, and punctuation tokens.<n>Despite their ubiquity, the function, semantic role, and origin of attention sinks remain poorly understood.
  arXiv  Detail & Related papers  (2025-02-02T21:15:07Z)
• Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMs [77.66717051042032]
  Practitioners have consistently observed three puzzling phenomena in transformer-based large language models. These phenomena are characterized by certain so-called "sink tokens" receiving disproportionately high attention weights. We elucidate the mechanisms behind extreme-token phenomena.
  arXiv  Detail & Related papers  (2024-10-17T17:54:06Z)
• When Attention Sink Emerges in Language Models: An Empirical View [39.36282162213973]
  Language Models (LMs) assign significant attention to the first token, even if it is not semantically important.<n>This phenomenon has been widely adopted in applications such as streaming/long context generation, KV cache optimization, inference acceleration, model quantization, and others.<n>We first demonstrate that attention sinks exist universally in LMs with various inputs, even in small models.
  arXiv  Detail & Related papers  (2024-10-14T17:50:28Z)
• Guiding Visual Question Answering with Attention Priors [76.21671164766073]
  We propose to guide the attention mechanism using explicit linguistic-visual grounding. This grounding is derived by connecting structured linguistic concepts in the query to their referents among the visual objects. The resultant algorithm is capable of probing attention-based reasoning models, injecting relevant associative knowledge, and regulating the core reasoning process.
  arXiv  Detail & Related papers  (2022-05-25T09:53:47Z)
• Is Sparse Attention more Interpretable? [52.85910570651047]
  We investigate how sparsity affects our ability to use attention as an explainability tool. We find that only a weak relationship between inputs and co-indexed intermediate representations exists -- under sparse attention. We observe in this setting that inducing sparsity may make it less plausible that attention can be used as a tool for understanding model behavior.
  arXiv  Detail & Related papers  (2021-06-02T11:42:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.