Behind RoPE: How Does Causal Mask Encode Positional Information?

• URL: http://arxiv.org/abs/2509.21042v1
• Date: Thu, 25 Sep 2025 11:48:24 GMT
• Title: Behind RoPE: How Does Causal Mask Encode Positional Information?
• Abstract summary: We show that the causal mask can induce position-dependent patterns in attention scores.<n>We find that the interaction of causal mask and RoPE distorts RoPE's relative attention score patterns into non-relative ones.
• Score: 40.16173166190937
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: While explicit positional encodings such as RoPE are a primary source of positional information in Transformer decoders, the causal mask also provides positional information. In this work, we prove that the causal mask can induce position-dependent patterns in attention scores, even without parameters or causal dependency in the input. Our theoretical analysis indicates that the induced attention pattern tends to favor nearby query-key pairs, mirroring the behavior of common positional encodings. Empirical analysis confirms that trained models exhibit the same behavior, with learned parameters further amplifying these patterns. Notably, we found that the interaction of causal mask and RoPE distorts RoPE's relative attention score patterns into non-relative ones. We consistently observed this effect in modern large language models, suggesting the importance of considering the causal mask as a source of positional information alongside explicit positional encodings.

Related papers

• Frayed RoPE and Long Inputs: A Geometric Perspective [3.573191378087653]
  Rotary Positional Embedding (RoPE) is a widely adopted technique for encoding position in language models.<n>We show that attention induces tight clustering of separated key and query latent point clouds, allowing for creation of sink tokens.<n>We propose RoPE-ID (In Distribution), a modification that allows attention layers to generalize to longer inputs out of the box.
  arXiv  Detail & Related papers  (2026-02-24T23:03:47Z)
• Untwisting RoPE: Frequency Control for Shared Attention in DiTs [84.14005261938284]
  Positional encodings are essential to transformer-based generative models.<n>We show that Rotary Positional Embeddings (RoPE) naturally decomposes into frequency components with distinct positional sensitivities.<n>We introduce a method for selectively modulating RoPE frequency bands so that attention reflects semantic similarity rather than strict positional alignment.
  arXiv  Detail & Related papers  (2026-02-04T20:01:59Z)
• Demystifying the Slash Pattern in Attention: The Role of RoPE [108.78812285352507]
  We demystify the emergence of Slash-Dominant Heads (SDHs) from both empirical and theoretical perspectives.<n>We find that SDHs are intrinsic to models and generalize to out-of-distribution prompts.
  arXiv  Detail & Related papers  (2026-01-13T07:40:57Z)
• DoPE: Denoising Rotary Position Embedding [60.779039511252584]
  Rotary Position Embedding (RoPE) in Transformer models has inherent limits that weaken length.<n>We reinterpret the attention map with positional encoding as a noisy feature map, and propose Denoising Positional extrapolation page (DoPE)<n>DoPE is a training-free method based on truncated matrix entropy to detect outlier frequency bands in the feature map.
  arXiv  Detail & Related papers  (2025-11-12T09:32:35Z)
• Do traveling waves make good positional encodings? [44.55744608160896]
  We propose RollPE, a novel positional encoding mechanism based on traveling waves.<n>We show it significantly outperforms traditional absolute positional embeddings.<n>We derive a mathematical equivalence of RollPE to a particular configuration of RoPE.
  arXiv  Detail & Related papers  (2025-11-11T14:32:45Z)
• Decoupling Positional and Symbolic Attention Behavior in Transformers [40.15480480887269]
  In Transformers, positional information is encoded using Positional s (PEs)<n>Recently, it has been argued that part of RoPE's success emerges from its ability to encode robust positional and semantic information using large and small frequencies.
  arXiv  Detail & Related papers  (2025-10-03T01:31:15Z)
• Sensitivity Meets Sparsity: The Impact of Extremely Sparse Parameter Patterns on Theory-of-Mind of Large Language Models [55.46269953415811]
  We identify ToM-sensitive parameters and show that perturbing as little as 0.001% of these parameters significantly degrades ToM performance.<n>Our results have implications for enhancing model alignment, mitigating biases, and improving AI systems designed for human interaction.
  arXiv  Detail & Related papers  (2025-04-05T17:45:42Z)
• Rotary Offset Features in Large Language Models [0.9137554315375919]
  We study the features and patterns that emerge in queries and keys when using rotary embeddings.<n>We derive bounds predicting which rotary frequencies give rise to rotary offset features.<n>We verify our predictions empirically across models of different sizes and architectures.
  arXiv  Detail & Related papers  (2025-03-03T18:55:09Z)
• On the Emergence of Position Bias in Transformers [59.87743433861665]
  This paper presents a graph-theoretic framework for analyzing position biases in multilayer positions.<n>Our framework offers a principled foundation for understanding positional interplay in transformers.
  arXiv  Detail & Related papers  (2025-02-04T02:53:07Z)
• Eliminating Position Bias of Language Models: A Mechanistic Approach [119.34143323054143]
  Position bias has proven to be a prevalent issue of modern language models (LMs)<n>Our mechanistic analysis attributes the position bias to two components employed in nearly all state-of-the-art LMs: causal attention and relative positional encodings.<n>By eliminating position bias, models achieve better performance and reliability in downstream tasks, including LM-as-a-judge, retrieval-augmented QA, molecule generation, and math reasoning.
  arXiv  Detail & Related papers  (2024-07-01T09:06:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.