Positional versus Symbolic Attention Heads: Learning Dynamics, RoPE Geometry, and Length Generalization

• URL: http://arxiv.org/abs/2605.31558v1
• Date: Fri, 29 May 2026 17:22:04 GMT
• Title: Positional versus Symbolic Attention Heads: Learning Dynamics, RoPE Geometry, and Length Generalization
• Abstract summary: We study the learning dynamics of attention heads in a controlled setting.<n>We show that successful learning is associated with the emergence of pure heads that express themselves as either positional or symbolic.
• Score: 1.5085587910317413
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Transformer-based language models are widespread in today's society. As such, understanding the mechanisms by which they solve structured tasks and predicting how they may behave in novel scenarios is of great importance for safe deployment. We study the learning dynamics of attention heads in a controlled setting by training a decoder-only Transformer (GPT-J) on two structurally equivalent multi-hop reasoning tasks: a number task requiring positional reasoning and a letter task requiring symbolic reasoning. Using a recently introduced metric that classifies attention-head behavior as positional or symbolic for a given prompt, we show that successful learning is associated with the emergence of pure heads, i.e., heads that express themselves as either positional or symbolic. Despite the tasks' structural equivalence, they impose different mechanistic demands: the number task requires both positional and symbolic heads, whereas the letter task requires only symbolic heads. We then identify the computational roles of these heads, characterize the basic functions they implement, and give theoretical constructions showing how single-layer RoPE-based attention can realize these functions through geometrically interpretable query, key, and value operations. This analysis yields a quantitative separation between positional and symbolic mechanisms in their robustness to longer sequences, formalized through a novel notion of discrepancy. We empirically validate the resulting predictions in both controlled and real-world models, showing that symbolic mechanisms extrapolate more reliably to longer sequences while positional mechanisms face sharper limitations.

Related papers

• How LLMs Follow Instructions: Skillful Coordination, Not a Universal Mechanism [2.5997274006052544]
  We investigate whether instruction-following relies on a universal mechanism or compositional skill deployment.<n>General probes trained across all tasks consistently underperform task-specific specialists, indicating limited representational sharing.<n> temporal analysis shows constraint satisfaction operates as dynamic monitoring during generation rather than pre-generation planning.
  arXiv  Detail & Related papers  (2026-04-07T16:12:52Z)
• Cognitive Mismatch in Multimodal Large Language Models for Discrete Symbol Understanding [96.81411333150213]
  This paper introduces a benchmark to evaluate how top-tier MLLMs navigate discrete semantic spaces.<n>Our investigation uncovers a counterintuitive phenomenon: models often fail at basic symbol recognition yet succeed in complex reasoning tasks.<n>This work offers a roadmap for developing more rigorous, human-aligned intelligent systems.
  arXiv  Detail & Related papers  (2026-03-19T04:08:20Z)
• Cognitive Mirrors: Exploring the Diverse Functional Roles of Attention Heads in LLM Reasoning [54.12174882424842]
  Large language models (LLMs) have achieved state-of-the-art performance in a variety of tasks, but remain largely opaque in terms of their internal mechanisms.<n>We propose a novel interpretability framework to systematically analyze the roles and behaviors of attention heads.
  arXiv  Detail & Related papers  (2025-12-03T10:24:34Z)
• Selection, Reflection and Self-Refinement: Revisit Reasoning Tasks via a Causal Lens [19.316594303998667]
  Reasoning tasks have long been regarded as rigorous benchmarks for assessing the capabilities of machine learning models.<n>We revisit reasoning tasks from a causal perspective, seeking to understand their behavior in latent space.<n>We introduce a framework, called SR$2$, that incorporates the estimated latent variables as feedback into the selection mechanism.
  arXiv  Detail & Related papers  (2025-10-09T13:45:31Z)
• Decoupling Positional and Symbolic Attention Behavior in Transformers [40.15480480887269]
  In Transformers, positional information is encoded using Positional s (PEs)<n>Recently, it has been argued that part of RoPE's success emerges from its ability to encode robust positional and semantic information using large and small frequencies.
  arXiv  Detail & Related papers  (2025-10-03T01:31:15Z)
• Multi-head Transformers Provably Learn Symbolic Multi-step Reasoning via Gradient Descent [66.78052387054593]
  This work investigates how transformers learn to solve symbolic multi-step reasoning problems through chain-of-thought processes.<n>We analyze two intertwined tasks: a backward reasoning task, where the model outputs a path from a goal node to the root, and a more complex forward reasoning task.<n>We show that trained one-layer transformers can provably solve both tasks with generalization guarantees to unseen trees.
  arXiv  Detail & Related papers  (2025-08-11T17:40:47Z)
• How do Transformers Learn Implicit Reasoning? [67.02072851088637]
  We study how implicit multi-hop reasoning emerges by training transformers from scratch in a controlled symbolic environment.<n>We find that training with atomic triples is not necessary but accelerates learning, and that second-hop generalization relies on query-level exposure to specific compositional structures.
  arXiv  Detail & Related papers  (2025-05-29T17:02:49Z)
• Iteration Head: A Mechanistic Study of Chain-of-Thought [6.072247578478243]
  Chain-of-Thought (CoT) reasoning is known to improve Large Language Models. This paper shows how CoT reasoning emerges in transformers in a controlled and interpretable setting.
  arXiv  Detail & Related papers  (2024-06-04T09:11:46Z)
• Compositional Attention: Disentangling Search and Retrieval [66.7108739597771]
  Multi-head, key-value attention is the backbone of the Transformer model and its variants. Standard attention heads learn a rigid mapping between search and retrieval. We propose a novel attention mechanism, called Compositional Attention, that replaces the standard head structure.
  arXiv  Detail & Related papers  (2021-10-18T15:47:38Z)
• The heads hypothesis: A unifying statistical approach towards understanding multi-headed attention in BERT [18.13834903235249]
  Multi-headed attention heads are a mainstay in transformer-based models. Different methods have been proposed to classify the role of each attention head based on the relations between tokens which have high pair-wise attention. We formalize a simple yet effective score that generalizes to all the roles of attention heads and employs hypothesis testing on this score for robust inference.
  arXiv  Detail & Related papers  (2021-01-22T14:10:59Z)
• CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning [138.40338621974954]
  CausalWorld is a benchmark for causal structure and transfer learning in a robotic manipulation environment. Tasks consist of constructing 3D shapes from a given set of blocks - inspired by how children learn to build complex structures.
  arXiv  Detail & Related papers  (2020-10-08T23:01:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.