AdaRoPE: Not All Attention Heads Should Rotate and Scale Equally

• URL: http://arxiv.org/abs/2607.19363v1
• Date: Fri, 05 Jun 2026 05:09:15 GMT
• Title: AdaRoPE: Not All Attention Heads Should Rotate and Scale Equally
• Abstract summary: We show that heads with different functional roles require distinct frequency ranges and attention scaling factors to operate effectively.<n>We propose AdaRoPE, which equips each attention head with learnable rotation frequencies and attention scaling factors.
• Score: 6.700920459684458
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Rotary Position Embedding (RoPE) is widely adopted in Transformers to encode positional information, yet standard implementations enforce a uniform frequency schedule and scaling across all attention heads. Using simplified retrieval tasks and length generalization scenarios, we show -- both empirically and theoretically -- that heads with different functional roles require distinct frequency ranges and attention scaling factors to operate effectively. Ignoring this structure leads to suboptimal utilization of embedding dimensions and degraded performance, particularly under long-context settings. To address these limitations, we propose AdaRoPE, which equips each attention head with learnable rotation frequencies and attention scaling factors. Pretrained LLMs with AdaRoPE consistently outperform existing RoPE variants, including partial RoPE and NoPE baselines. For context extension, we further show that uniform frequency and attention scaling, used in methods such as YaRN, are suboptimal. By applying head-specific scaling, AdaRoPE enables better context extension while better preserving short-context performance in both the extrapolation setting and the long-context continued pretraining setting. These results highlight the importance of optimizing rotary position embedding at the level of individual attention heads.

Related papers

• Disentangling the Expressivity of RoPE [53.85490969529292]
  We formalize accounts for fully uniform, finite-precision soft-attention transformers.<n>We find that, if every rotary component is periodic, RoPE transformers recognize exactly the languages definable in past temporal logic with modular predicates.
  arXiv  Detail & Related papers  (2026-08-12T10:37:21Z)
• LeRoPE: Learnable RoPE Frequencies Improve Language Modeling [20.518459444800474]
  We observe and analyze the emergence of a high-norm, positional LeRoPE band.<n>LeRoPE consistently outperforms RoPE and partial RoPE across all scales.
  arXiv  Detail & Related papers  (2026-07-11T05:47:23Z)
• Beyond Real: Imaginary Extension of Rotary Position Embeddings for Long-Context LLMs [72.8830548005884]
  Rotary Position Embeddings (RoPE) have become a standard for encoding sequence order in Large Language Models.<n>Standard implementations utilize only the real component of the complex-valued dot product for attention score calculation.<n>We propose an extension that re-incorporates this imaginary component.
  arXiv  Detail & Related papers  (2025-12-08T12:59:54Z)
• RoSA: Enhancing Parameter-Efficient Fine-Tuning via RoPE-aware Selective Adaptation in Large Language Models [23.726452130486496]
  Fine-tuning large language models is essential for task-specific adaptation, yet it remains computationally prohibitive.<n>We propose RoPE-aware Selective Adaptation (RoSA), a novel PEFT framework that allocates trainable parameters in a more targeted and effective manner.<n>RoSA comprises a RoPE-aware Attention Enhancement (RoAE) module, and a Dynamic Layer Selection (DLS) strategy that adaptively identifies and updates the most critical layers based on LayerNorm norms.
  arXiv  Detail & Related papers  (2025-11-21T09:55:01Z)
• DoPE: Denoising Rotary Position Embedding [60.779039511252584]
  Rotary Position Embedding (RoPE) in Transformer models has inherent limits that weaken length.<n>We reinterpret the attention map with positional encoding as a noisy feature map, and propose Denoising Positional extrapolation page (DoPE)<n>DoPE is a training-free method based on truncated matrix entropy to detect outlier frequency bands in the feature map.
  arXiv  Detail & Related papers  (2025-11-12T09:32:35Z)
• Positional Encoding via Token-Aware Phase Attention [45.855203550592734]
  We show that Rotary Positional Embedding (RoPE) introduces an intrinsic distance-dependent bias in attention scores that limits RoPE's ability to model long-context.<n>This paper introduces Token-Aware Phase Attention (TAPA), a new positional encoding method that incorporates a learnable phase function into the attention mechanism.
  arXiv  Detail & Related papers  (2025-09-16T03:53:32Z)
• Context-aware Rotary Position Embedding [0.0]
  Rotary Positional Embeddings (RoPE) have become a widely adopted solution due to their compatibility with relative position encoding and computational efficiency.<n>We propose CARoPE (Context-Aware Rotary Positional Embedding), a novel generalization of RoPE that dynamically generates head-specific frequency patterns conditioned on token embeddings.<n>CaroPE consistently outperforms RoPE and other common positional encoding baselines, achieving significantly lower perplexity, even at longer context lengths.
  arXiv  Detail & Related papers  (2025-07-30T20:32:19Z)
• Fourier Position Embedding: Enhancing Attention's Periodic Extension for Length Generalization [23.936687072300053]
  We show that Rotary Position Embedding (RoPE) enables periodic attention by implicitly achieving Non-Uniform Discrete Fourier Transform.<n>We propose Fourier Position Embedding (FoPE) which enhances attention's frequency-domain properties to improve both its periodic extension and length generalization.
  arXiv  Detail & Related papers  (2024-12-23T17:44:01Z)
• Scaling Laws of RoPE-based Extrapolation [103.33995311915864]
  We propose textbftextitScaling Laws of RoPE-based Extrapolation to describe the relationship between the extrapolation performance and base value. We achieve extrapolation up to 1 million context length within only 16K training length on LLaMA2 7B and 13B.
  arXiv  Detail & Related papers  (2023-10-08T15:50:36Z)
• The Impact of Positional Encoding on Length Generalization in Transformers [50.48278691801413]
  We compare the length generalization performance of decoder-only Transformers with five different position encoding approaches. Our findings reveal that the most commonly used positional encoding methods, such as ALiBi, Rotary, and APE, are not well suited for length generalization in downstream tasks.
  arXiv  Detail & Related papers  (2023-05-31T00:29:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.