Decoupling Positional and Symbolic Attention Behavior in Transformers

• URL: http://arxiv.org/abs/2511.11579v1
• Date: Fri, 03 Oct 2025 01:31:15 GMT
• Title: Decoupling Positional and Symbolic Attention Behavior in Transformers
• Authors: Felipe Urrutia, Jorge Salas, Alexander Kozachinskiy, Cristian Buc Calderon, Hector Pasten, Cristobal Rojas,
• Abstract summary: In Transformers, positional information is encoded using Positional s (PEs)<n>Recently, it has been argued that part of RoPE's success emerges from its ability to encode robust positional and semantic information using large and small frequencies.
• Score: 40.15480480887269
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: An important aspect subtending language understanding and production is the ability to independently encode positional and symbolic information of the words within a sentence. In Transformers, positional information is typically encoded using Positional Encodings (PEs). One such popular PE, namely Rotary PE (RoPE), has been widely used due to its empirical success. Recently, it has been argued that part of RoPE's success emerges from its ability to encode robust positional and semantic information using large and small frequencies, respectively. In this work, we perform a deeper dive into the positional versus symbolic dichotomy of attention heads behavior, both at the theoretical and empirical level. We provide general definitions of what it means for a head to behave positionally or symbolically, prove that these are two mutually exclusive behaviors and develop a metric to quantify them. We apply our framework to analyze Transformer-based LLMs using RoPE and find that all heads exhibit a strong correspondence between behavior and frequency use. Finally, we introduce canonical tasks designed to be either purely positional or symbolic, and demonstrate that the Transformer performance causally relates to the ability of attention heads to leverage the appropriate frequencies. In particular, we show that we can control the Transformer performance by controlling which frequencies the attention heads can access. Altogether, our work provides a detailed understanding of RoPE, and how its properties relate to model behavior.

Related papers

• Untwisting RoPE: Frequency Control for Shared Attention in DiTs [84.14005261938284]
  Positional encodings are essential to transformer-based generative models.<n>We show that Rotary Positional Embeddings (RoPE) naturally decomposes into frequency components with distinct positional sensitivities.<n>We introduce a method for selectively modulating RoPE frequency bands so that attention reflects semantic similarity rather than strict positional alignment.
  arXiv  Detail & Related papers  (2026-02-04T20:01:59Z)
• Do traveling waves make good positional encodings? [44.55744608160896]
  We propose RollPE, a novel positional encoding mechanism based on traveling waves.<n>We show it significantly outperforms traditional absolute positional embeddings.<n>We derive a mathematical equivalence of RollPE to a particular configuration of RoPE.
  arXiv  Detail & Related papers  (2025-11-11T14:32:45Z)
• Context-aware Rotary Position Embedding [0.0]
  Rotary Positional Embeddings (RoPE) have become a widely adopted solution due to their compatibility with relative position encoding and computational efficiency.<n>We propose CARoPE (Context-Aware Rotary Positional Embedding), a novel generalization of RoPE that dynamically generates head-specific frequency patterns conditioned on token embeddings.<n>CaroPE consistently outperforms RoPE and other common positional encoding baselines, achieving significantly lower perplexity, even at longer context lengths.
  arXiv  Detail & Related papers  (2025-07-30T20:32:19Z)
• Revisiting LRP: Positional Attribution as the Missing Ingredient for Transformer Explainability [53.21677928601684]
  Layer-wise relevance propagation is one of the most promising approaches to explainability in deep learning.<n>We propose specialized theoretically-grounded LRP rules designed to propagate attributions across various positional encoding methods.<n>Our method significantly outperforms the state-of-the-art in both vision and NLP explainability tasks.
  arXiv  Detail & Related papers  (2025-06-02T18:07:55Z)
• PaTH Attention: Position Encoding via Accumulating Householder Transformations [56.32365080761523]
  PaTH is a flexible data-dependent position encoding scheme based on accumulated products of Householder transformations.<n>We derive an efficient parallel algorithm for training through exploiting a compact representation of products of Householder matrices.
  arXiv  Detail & Related papers  (2025-05-22T08:36:09Z)
• DAPE V2: Process Attention Score as Feature Map for Length Extrapolation [63.87956583202729]
  We conceptualize attention as a feature map and apply the convolution operator to mimic the processing methods in computer vision. The novel insight, which can be adapted to various attention-related models, reveals that the current Transformer architecture has the potential for further evolution.
  arXiv  Detail & Related papers  (2024-10-07T07:21:49Z)
• Learning interpretable positional encodings in transformers depends on initialization [14.732076081683418]
  positional encoding (PE) provides essential information that distinguishes the position and order amongst tokens in a sequence.<n>We show that the choice of a learnable PE greatly influences its ability to learn interpretable PEs.<n>We find that a learned PE from a small-norm distribution can uncover interpretable PEs that mirror ground truth positions in multiple dimensions.
  arXiv  Detail & Related papers  (2024-06-12T14:37:29Z)
• Contextual Counting: A Mechanistic Study of Transformers on a Quantitative Task [40.85615657802704]
  This paper introduces the contextual counting task, a novel toy problem aimed at enhancing our understanding of Transformers. We present theoretical and empirical analysis using both causal and non-causal Transformer architectures.
  arXiv  Detail & Related papers  (2024-05-30T20:52:23Z)
• The Impact of Positional Encoding on Length Generalization in Transformers [50.48278691801413]
  We compare the length generalization performance of decoder-only Transformers with five different position encoding approaches. Our findings reveal that the most commonly used positional encoding methods, such as ALiBi, Rotary, and APE, are not well suited for length generalization in downstream tasks.
  arXiv  Detail & Related papers  (2023-05-31T00:29:55Z)
• Your Transformer May Not be as Powerful as You Expect [88.11364619182773]
  We mathematically analyze the power of RPE-based Transformers regarding whether the model is capable of approximating any continuous sequence-to-sequence functions. We present a negative result by showing there exist continuous sequence-to-sequence functions that RPE-based Transformers cannot approximate no matter how deep and wide the neural network is. We develop a novel attention module, called Universal RPE-based (URPE) Attention, which satisfies the conditions.
  arXiv  Detail & Related papers  (2022-05-26T14:51:30Z)
• Conformer-based End-to-end Speech Recognition With Rotary Position Embedding [11.428057887454008]
  We introduce rotary position embedding (RoPE) in the convolution-augmented transformer (conformer) RoPE encodes absolute positional information into the input sequence by a rotation matrix, and then naturally incorporates explicit relative position information into a self-attention module. Our model achieves a relative word error rate reduction of 8.70% and 7.27% over the conformer on test-clean and test-other sets of the LibriSpeech corpus respectively.
  arXiv  Detail & Related papers  (2021-07-13T08:07:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.