How Data Shapes RoPE Frequency Usage: From Positional Scale Matching to Length Generalization

• URL: http://arxiv.org/abs/2607.07678v1
• Date: Wed, 08 Jul 2026 17:38:14 GMT
• Title: How Data Shapes RoPE Frequency Usage: From Positional Scale Matching to Length Generalization
• Abstract summary: Rotary Position Embeddings (RoPE) provide transformers with a fixed grid of positional frequencies.<n>We study what determines this frequency usage and propose a data-centered explanation.
• Score: 19.24080183297062
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Rotary Position Embeddings (RoPE) provide transformers with a fixed grid of positional frequencies, yet trained models use these frequencies highly non-uniformly. We study what determines this frequency usage and propose a data-centered explanation: RoPE frequencies are selected to match the relative-distance structure of the training data. Viewing each frequency as a positional lens, we formalize a field-resolution tradeoff and show that, for a data-induced dependency profile of width $W$, the optimal frequency scales as $1/W$. This frequency-matching principle explains controlled observations on synthetic and text-based data, and suggests that the mid-low frequency bands observed in language models arise from the multi-scale dependency structure of natural language. We further connect frequency selection to position-interpolation-based length generalization: scaling frequencies down expands the effective field while reducing resolution. This helps when longer-context dependencies are approximate dilations of those seen during training, but can fail when relevant dependencies do not scale with context length. Empirically, we show that natural language exhibits approximate self-similarity across positional scales, explaining why test-time frequency scaling can support long-context generalization. Overall, our results identify a data-driven mechanism behind emergent RoPE frequency usage and show that long-context generalization depends on two forms of scale matching: between learned frequencies and training-time dependencies, and between frequency scaling and how those dependencies extend to longer contexts.

Related papers

• Faster 3D Gaussian Splatting Convergence via Structure-Aware Densification [52.517763621139956]
  3D Gaussian Splatting has emerged as a powerful scene representation for real-time novel-view synthesis.<n>Standard adaptive density control relies on screen-space positional gradients.<n>We present a structure-aware densification framework for 3D Gaussian Splatting.
  arXiv  Detail & Related papers  (2026-04-30T15:37:20Z)
• Corpus Frequencies in Morphological Inflection: Do They Matter? [0.5934049163128877]
  We explore the incorporation of corpus frequency information into the task of morphological inflection along three key dimensions during system development.<n>We show that frequency-aware training outperforms uniform sampling in 26 out of 43 languages.
  arXiv  Detail & Related papers  (2025-10-27T09:12:04Z)
• Stationarity Exploration for Multivariate Time Series Forecasting [16.66747488602513]
  We propose a simple yet effective Amplitude-Phase Reconstruct Network (APRNet)<n>APRNet models the inter-relationships of amplitude and phase, which prevents the amplitude and phase from being constrained by different physical quantities.<n>We propose a novel Kolmogorov-Arnold-Network-based Local Correlation (KLC) module to adaptively fit local functions.
  arXiv  Detail & Related papers  (2025-08-12T13:15:51Z)
• TFKAN: Time-Frequency KAN for Long-Term Time Series Forecasting [16.937153556835423]
  Kolmogorov-Arnold Networks (KANs) are highly effective in long-term time series forecasting.<n>We propose the $textbfT$ime-$textbfF$requency KAN (TFKAN) to integrate information from both time and frequency domains.<n> Experimental results demonstrate that TFKAN consistently outperforms state-of-the-art (SOTA) methods across multiple datasets.
  arXiv  Detail & Related papers  (2025-06-15T02:53:25Z)
• The Harmonic Structure of Information Contours [54.38365999922221]
  We find consistent evidence of periodic patterns in information rate in texts in English, Spanish, German, Dutch, Basque, and Brazilian Portuguese.<n>Many dominant frequencies align with discourse structure, suggesting these oscillations reflect meaningful linguistic organization.
  arXiv  Detail & Related papers  (2025-06-04T12:56:30Z)
• FreRA: A Frequency-Refined Augmentation for Contrastive Learning on Time Series Classification [56.925103708982164]
  We present a novel perspective from the frequency domain and identify three advantages for downstream classification: global, independent, and compact.<n>We propose the lightweight yet effective Frequency Refined Augmentation (FreRA) tailored for time series contrastive learning on classification tasks.<n>FreRA consistently outperforms ten leading baselines on time series classification, anomaly detection, and transfer learning tasks.
  arXiv  Detail & Related papers  (2025-05-29T07:18:28Z)
• Freqformer: Image-Demoiréing Transformer via Efficient Frequency Decomposition [83.40450475728792]
  We present Freqformer, a Transformer-based framework specifically designed for image demoir'eing through targeted frequency separation.<n>Our method performs an effective frequency decomposition that explicitly splits moir'e patterns into high-frequency spatially-localized textures and low-frequency scale-robust color distortions.<n>Experiments on various demoir'eing benchmarks demonstrate that Freqformer achieves state-of-the-art performance with a compact model size.
  arXiv  Detail & Related papers  (2025-05-25T12:23:10Z)
• Multi-View Frequency-Attention Alternative to CNN Frontends for Automatic Speech Recognition [12.980843126905203]
  We show that global attention over frequencies is beneficial over local convolution. We obtain 2.4 % relative word error rate reduction on a production scale replacing its convolutional neural network transducer.
  arXiv  Detail & Related papers  (2023-06-12T08:37:36Z)
• Adaptive Frequency Learning in Two-branch Face Forgery Detection [66.91715092251258]
  We propose Adaptively learn Frequency information in the two-branch Detection framework, dubbed AFD. We liberate our network from the fixed frequency transforms, and achieve better performance with our data- and task-dependent transform layers.
  arXiv  Detail & Related papers  (2022-03-27T14:25:52Z)
• Transforming Spectrum and Prosody for Emotional Voice Conversion with Non-Parallel Training Data [91.92456020841438]
  Many studies require parallel speech data between different emotional patterns, which is not practical in real life. We propose a CycleGAN network to find an optimal pseudo pair from non-parallel training data. We also study the use of continuous wavelet transform (CWT) to decompose F0 into ten temporal scales, that describes speech prosody at different time resolution.
  arXiv  Detail & Related papers  (2020-02-01T12:36:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.