Demystifying the Slash Pattern in Attention: The Role of RoPE

• URL: http://arxiv.org/abs/2601.08297v1
• Date: Tue, 13 Jan 2026 07:40:57 GMT
• Title: Demystifying the Slash Pattern in Attention: The Role of RoPE
• Abstract summary: We demystify the emergence of Slash-Dominant Heads (SDHs) from both empirical and theoretical perspectives.<n>We find that SDHs are intrinsic to models and generalize to out-of-distribution prompts.
• Score: 108.78812285352507
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Language Models (LLMs) often exhibit slash attention patterns, where attention scores concentrate along the $Δ$-th sub-diagonal for some offset $Δ$. These patterns play a key role in passing information across tokens. But why do they emerge? In this paper, we demystify the emergence of these Slash-Dominant Heads (SDHs) from both empirical and theoretical perspectives. First, by analyzing open-source LLMs, we find that SDHs are intrinsic to models and generalize to out-of-distribution prompts. To explain the intrinsic emergence, we analyze the queries, keys, and Rotary Position Embedding (RoPE), which jointly determine attention scores. Our empirical analysis reveals two characteristic conditions of SDHs: (1) Queries and keys are almost rank-one, and (2) RoPE is dominated by medium- and high-frequency components. Under these conditions, queries and keys are nearly identical across tokens, and interactions between medium- and high-frequency components of RoPE give rise to SDHs. Beyond empirical evidence, we theoretically show that these conditions are sufficient to ensure the emergence of SDHs by formalizing them as our modeling assumptions. Particularly, we analyze the training dynamics of a shallow Transformer equipped with RoPE under these conditions, and prove that models trained via gradient descent exhibit SDHs. The SDHs generalize to out-of-distribution prompts.

Related papers

• Sparsity as a Key: Unlocking New Insights from Latent Structures for Out-of-Distribution Detection [4.496267803613718]
  We present the first application of SAEs to the ViT [] token for out-of-distribution (OOD) detection.<n>Our findings demonstrate that the sparse, disentangled features revealed by SAEs can serve as a powerful, interpretable tool for robust OOD detection in vision models.
  arXiv  Detail & Related papers  (2026-04-29T08:23:38Z)
• Frayed RoPE and Long Inputs: A Geometric Perspective [3.573191378087653]
  Rotary Positional Embedding (RoPE) is a widely adopted technique for encoding position in language models.<n>We show that attention induces tight clustering of separated key and query latent point clouds, allowing for creation of sink tokens.<n>We propose RoPE-ID (In Distribution), a modification that allows attention layers to generalize to longer inputs out of the box.
  arXiv  Detail & Related papers  (2026-02-24T23:03:47Z)
• Why Attention Patterns Exist: A Unifying Temporal Perspective Analysis [61.597286699809395]
  We introduce textbfTemporal Attention Pattern Predictability Analysis (TAPPA)<n>TAPPA characterizes attention patterns as predictable patterns with clear regularities and unpredictable patterns that appear effectively random.<n>We provide a detailed mathematical analysis of three representative cases through the joint effect of queries, keys, and Rotary Positional Embeddings (RoPE)
  arXiv  Detail & Related papers  (2026-01-29T13:40:23Z)
• Consistency Is Not Always Correct: Towards Understanding the Role of Exploration in Post-Training Reasoning [75.79451512757844]
  Foundation models exhibit broad knowledge but limited task-specific reasoning.<n> RLVR and inference scaling motivate post-training strategies such as RLVR and inference scaling.<n>We show that RLVR induces a squeezing effect, reducing reasoning entropy and forgetting some correct paths.
  arXiv  Detail & Related papers  (2025-11-10T18:25:26Z)
• From Harm to Help: Turning Reasoning In-Context Demos into Assets for Reasoning LMs [58.02809208460186]
  We revisit this paradox using high-quality traces from DeepSeek-R1 as demonstrations.<n>We find that adding more exemplars consistently degrades accuracy, even when demonstrations are optimal.<n>We introduce Insight-to-solve (I2S), a sequential test-time procedure that turns demonstrations into explicit, reusable insights.
  arXiv  Detail & Related papers  (2025-09-27T08:59:31Z)
• Behind RoPE: How Does Causal Mask Encode Positional Information? [40.16173166190937]
  We show that the causal mask can induce position-dependent patterns in attention scores.<n>We find that the interaction of causal mask and RoPE distorts RoPE's relative attention score patterns into non-relative ones.
  arXiv  Detail & Related papers  (2025-09-25T11:48:24Z)
• How do Transformers Learn Implicit Reasoning? [67.02072851088637]
  We study how implicit multi-hop reasoning emerges by training transformers from scratch in a controlled symbolic environment.<n>We find that training with atomic triples is not necessary but accelerates learning, and that second-hop generalization relies on query-level exposure to specific compositional structures.
  arXiv  Detail & Related papers  (2025-05-29T17:02:49Z)
• I Have Covered All the Bases Here: Interpreting Reasoning Features in Large Language Models via Sparse Autoencoders [8.1201445044499]
  Internal mechanisms behind reasoning in LLMs remain unexplored.<n>We employ Sparse Autoencoders to test our hypothesis.<n>Our work provides the first step towards a mechanistic understanding of reasoning in LLMs.
  arXiv  Detail & Related papers  (2025-03-24T16:54:26Z)
• Rotary Offset Features in Large Language Models [0.9137554315375919]
  We study the features and patterns that emerge in queries and keys when using rotary embeddings.<n>We derive bounds predicting which rotary frequencies give rise to rotary offset features.<n>We verify our predictions empirically across models of different sizes and architectures.
  arXiv  Detail & Related papers  (2025-03-03T18:55:09Z)
• Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understanding [58.364933651703524]
  We show that concentrated massive values consistently emerge in specific regions of attention queries.<n>These massive values play a critical role in interpreting contextual knowledge.<n>We trace the emergence of massive values and find that such concentration is caused by Rotary Positional.
  arXiv  Detail & Related papers  (2025-02-03T17:47:03Z)
• Interpreting Attention Layer Outputs with Sparse Autoencoders [3.201633659481912]
  Decomposing model activations into interpretable components is a key open problem in mechanistic interpretability. In this work we train SAEs on attention layer outputs and show that also here SAEs find a sparse, interpretable decomposition. We show that Sparse Autoencoders are a useful tool that enable researchers to explain model behavior in greater detail than prior work.
  arXiv  Detail & Related papers  (2024-06-25T17:43:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.