A Unifying View of Attention Sinks: Two Algorithms, Two Solutions

• URL: http://arxiv.org/abs/2606.08105v1
• Date: Sat, 06 Jun 2026 11:10:54 GMT
• Title: A Unifying View of Attention Sinks: Two Algorithms, Two Solutions
• Abstract summary: We show that visually similar sink patterns can reflect two distinct mechanisms.<n>i adaptive nop, where a head suppresses its update by routing to a null token, and ii broadcast, where a sink aggregates and redistributes global information.
• Score: 11.81228217746692
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: When attention concentrates on a single token, a sink, what is the model actually computing? Attention sinks are ubiquitous in softmax transformers, yet this shared visual signature can hide fundamentally different algorithms. We show that visually similar sink patterns can reflect two distinct mechanisms: {i} adaptive nop, where a head suppresses its update by routing to a null token, and {ii} broadcast, where a sink aggregates and redistributes global information. In that case, sinks serve an analogous role: a safe destination when there is nothing useful to compute. Proposed interventions like gating or registers work because they implicitly target one or the other, revealing a duality between method and assumed mechanism: gating implicitly assumes nop; registers implicitly assume broadcast. Each mechanism leaves distinct traces (nop sinks exhibit negligible value norms; broadcast sinks induce low-rank outputs) which we formalize on synthetic tasks and use to derive practical diagnostics. Applied to pretrained vision transformers, these diagnostics reveal that both mechanisms exist at scale: sinks transition from CLS in early layers to patches in deeper layers, and concentrate in specialized heads. Strikingly, register tokens, designed for broadcast, are repurposed to also serve nop, confirming that neither intervention alone suffices. Combining gating with registers yields complementary gains in stability and performance. Overall, we find that the same attention pattern can reflect two very different computations and effective intervention requires first asking what the model is actually computing.

Related papers

• Tracing distinguishability through transformer processing with stochastic LayerNorm [0.0]
  We give representations volume, turning similarity into statistical distinguishability.<n>We realize this idea in pretrained transformers through a light-touch modification to LayerNorm.<n>These results establish distinguishability as a functionally grounded lens on transformer computation.
  arXiv  Detail & Related papers  (2026-08-31T12:57:38Z)
• A Compositional Theory of Causally Masked Transformers [50.88598486616582]
  We develop a formalization that derives expressivity directly from the model's implemented dynamics.<n>Each attention head updates its own state independently within a layer.<n>A modified form of soft attention supports irreversible, checklist-like state.
  arXiv  Detail & Related papers  (2026-07-29T14:47:19Z)
• DUET -- Dual User Embedding Transformers for Offsite Conversion Prediction [8.972672149050627]
  Offsite conversion rate (OCVR) prediction is an important ranking problem in computational recommendation systems.<n>This task presents a modeling challenge: click signals are abundant and exhibit short temporal horizons, whereas conversion signals are inherently sparse, long-delayed, and frequently unattributed.<n>We propose DUET, a framework that explicitly partitions user behavioral data into two domain-coherent streams -- clicks and conversions.
  arXiv  Detail & Related papers  (2026-06-08T23:13:58Z)
• The Structural Origin of Attention Sink: Variance Discrepancy, Super Neurons, and Dimension Disparity [22.040532283910522]
  This work provides a textitmechanistic explanation for the phenomenon.<n>We trace its root to the value aggregation process inherent in self-attention.<n>As a proof of concept, we propose textithead-wise RMSNorm, an architectural modification that stabilizes value aggregation outputs during pre-training.
  arXiv  Detail & Related papers  (2026-05-07T17:28:55Z)
• Attention Sinks Are Provably Necessary in Softmax Transformers: Evidence from Trigger-Conditional Tasks [1.6921396880325779]
  We prove that computing a simple trigger-conditional behavior necessarily induces a sink in softmax self-attention models.<n>Our results formalize a familiar intuition: normalization over a probability simplex must force attention to collapse onto a stable anchor to realize a default state.<n>We also prove that non-normalized ReLU attention can solve the same task without any sink, confirming that the normalization constraint is the fundamental driver of sink behavior.
  arXiv  Detail & Related papers  (2026-03-12T03:13:28Z)
• How Attention Sinks Emerge in Large Language Models: An Interpretability Perspective [67.08306259095778]
  We identify a simple mechanism that enables the model to recognize token at position zero and induce an attention sink within two transformer blocks.<n>By analyzing training traces from a 30B A3B MoE model trained from scratch, we find that this mechanism emerges early in training and becomes increasingly concentrated in the first two layers.
  arXiv  Detail & Related papers  (2026-02-04T11:10:34Z)
• A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training [86.64715217940274]
  Outliers function jointly with normalization.<n>Outliers serve more as rescale factors rather than contributors.<n>Outliers can be absorbed into learnable parameters or mitigated via explicit gated rescaling.
  arXiv  Detail & Related papers  (2026-01-30T13:29:45Z)
• Attention Projection Mixing with Exogenous Anchors [0.0]
  Cross-layer reuse of early attention projections can improve data efficiency, but it creates a structural conflict.<n>We show this conflict is a hidden limiter of internal-anchor designs.<n>We propose ExoFormer, which resolves the conflict by learning anchor projections outside the sequential layer stack.
  arXiv  Detail & Related papers  (2026-01-13T01:52:19Z)
• A Unified Cortical Circuit Model with Divisive Normalization and Self-Excitation for Robust Representation and Memory Maintenance [2.705743343600661]
  We introduce a recurrent neural circuit that combines divisive normalization with self-excitation to achieve robust encoding.<n>We demonstrate the model's versatility in two canonical tasks.<n>This work establishes a unified mathematical framework that bridges noise suppression, working memory, and approximate Bayesian inference.
  arXiv  Detail & Related papers  (2025-08-18T08:00:24Z)
• Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformers [54.20763128054692]
  We study how a two-attention-layer transformer is trained to perform ICL on $n$-gram Markov chain data. We prove that the gradient flow with respect to a cross-entropy ICL loss converges to a limiting model.
  arXiv  Detail & Related papers  (2024-09-09T18:10:26Z)
• An Analysis of Attention via the Lens of Exchangeability and Latent Variable Models [64.87562101662952]
  We show that input tokens are often exchangeable since they already include positional encodings. We establish the existence of a sufficient and minimal representation of input tokens. We prove that attention with the desired parameter infers the latent posterior up to an approximation error.
  arXiv  Detail & Related papers  (2022-12-30T17:59:01Z)
• Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth [48.16156149749371]
  This work proposes a new way to understand self-attention networks. We show that their output can be decomposed into a sum of smaller terms. We prove that self-attention possesses a strong inductive bias towards "token"
  arXiv  Detail & Related papers  (2021-03-05T00:39:05Z)
• Attention that does not Explain Away [54.42960937271612]
  Models based on the Transformer architecture have achieved better accuracy than the ones based on competing architectures for a large set of tasks. A unique feature of the Transformer is its universal application of a self-attention mechanism, which allows for free information flow at arbitrary distances. We propose a doubly-normalized attention scheme that is simple to implement and provides theoretical guarantees for avoiding the "explaining away" effect.
  arXiv  Detail & Related papers  (2020-09-29T21:05:39Z)
• Attentive WaveBlock: Complementarity-enhanced Mutual Networks for Unsupervised Domain Adaptation in Person Re-identification and Beyond [97.25179345878443]
  This paper proposes a novel light-weight module, the Attentive WaveBlock (AWB) AWB can be integrated into the dual networks of mutual learning to enhance the complementarity and further depress noise in the pseudo-labels. Experiments demonstrate that the proposed method achieves state-of-the-art performance with significant improvements on multiple UDA person re-identification tasks.
  arXiv  Detail & Related papers  (2020-06-11T15:40:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.