Content-Based Addressing for Long Context

• URL: http://arxiv.org/abs/2609.07314v2
• Date: Sun, 13 Sep 2026 21:15:00 GMT
• Title: Content-Based Addressing for Long Context
• Abstract summary: Rotary position embedding (RoPE) uses each token's integer position to determine the rotation applied inside attention.<n>This works well for local token order, but increasing context length creates a positional train-test mismatch.<n>We instead divide a token stream into units, retain ordinary RoPE positions within each unit, and assign every completed unit an address computed from its content.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Rotary position embedding (RoPE) uses each token's integer position to determine the rotation applied inside attention. This works well for local token order, but increasing context length creates a positional train-test mismatch: RoPE produces relative rotations at offsets not seen during training. Methods that rescale, interpolate, randomize, or bias positions specify how attention handles those offsets, but still derive positional information from a growing token counter. We instead divide a token stream into units, retain ordinary RoPE positions within each unit, and assign every completed unit an address computed from its content. Adding units then applies the same learned map to new content rather than extending a positional range or an identifier table. We prove that this construction preserves local RoPE exactly, leaves the attention comparison between two fixed tokens unchanged when other units are inserted or reordered, and does not create new relative rotations merely because more units are added. In a character-level Tiny Shakespeare diagnostic, all-token validation perplexity remains approximately constant from contexts of 256 to 4096 characters. A second diagnostic shows that content-based addressing can retrieve and use information from multiple serialized facts. These are controlled shallow experiments, not scale benchmarks, but they support a direct prescription: use position to address locally and content to address across units.

Related papers

• Position Encoding in Transformers: From Absolute and Relative Methods to Rotary Position Embeddings and Long-Context Scaling [3.3534582603188476]
  We develop a unified account of sinusoidal and learned absolute position embeddings, Shaw-style relative position representations, Transformer-XL, T5 relative position bias, ALiBi, and Rotary Position Embeddings (RoPE)<n>We then examine long-context extensions, including Position Interpolation, RoPE scaling laws, NTK-aware scaling, Dynamic NTK, NTK-by-parts, YaRN, LongRoPE, and LongRoPE2.<n>A central conclusion is that the ability to compute positional features beyond the training length does not imply reliable long-context generalization.
  arXiv  Detail & Related papers  (2026-08-09T10:15:49Z)
• RoVE: Rotary Value Embeddings Attention for Relative Position-dependent Value Pathways [52.94485753759119]
  Rotary Position Embeddings (RoPE) make attention scores position-relative but leave the value pathway position-blind.<n>We propose RoVE, a parameter-free modification that makes values position-sensitive by rotating them simultaneously with keys, and show that it turns RoPE attention into attentive convolution.
  arXiv  Detail & Related papers  (2026-06-09T09:56:50Z)
• SeqPE: Transformer with Sequential Position Encoding [76.22159277300891]
  SeqPE represents each $n$-dimensional position index as a symbolic sequence and employs a lightweight sequential position encoder to learn their embeddings.<n> Experiments across language modeling, long-context question answering, and 2D image classification demonstrate that SeqPE not only surpasses strong baselines in perplexity, exact match (EM) and accuracy--but also enables seamless generalization to multi-dimensional inputs without requiring manual architectural redesign.
  arXiv  Detail & Related papers  (2025-06-16T09:16:40Z)
• Circle-RoPE: Cone-like Decoupled Rotary Positional Embedding for Large Vision-Language Models [49.122200327049676]
  Rotary Position Embedding (RoPE) is a widely adopted technique for encoding relative positional information in large language models.<n>When extended to vision-language models (VLMs), RoPE and its variants enforce relative positional dependencies separately within text and image tokens.<n>We introduce Circle-RoPE, a novel encoding scheme designed to eliminate spurious cross-modal biases.
  arXiv  Detail & Related papers  (2025-05-22T09:05:01Z)
• PaTH Attention: Position Encoding via Accumulating Householder Transformations [56.32365080761523]
  PaTH is a flexible data-dependent position encoding scheme based on accumulated products of Householder transformations.<n>We derive an efficient parallel algorithm for training through exploiting a compact representation of products of Householder matrices.
  arXiv  Detail & Related papers  (2025-05-22T08:36:09Z)
• Benchmarking Rotary Position Embeddings for Automatic Speech Recognition [17.360059094663182]
  Relative Position (RelPos) embeddings are widely used in Automatic Speech Recognition (ASR)<n>In contrast, Rotary Positional Embedding (RoPE) rotates each input vector based on its absolute position, taking linear time to sequence length.<n>This work evaluates RoPE across diverse ASR tasks with training data ranging from 100 to 50,000 hours.
  arXiv  Detail & Related papers  (2025-01-10T15:30:46Z)
• When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Training [51.23520027773028]
  Extending context window sizes allows large language models to process longer sequences and handle more complex tasks. We observe that using RoPE with BFloat16 format results in numerical issues, causing it to deviate from its intended relative positional encoding. We develop AnchorAttention, a plug-and-play attention method that alleviates numerical issues caused by BFloat16.
  arXiv  Detail & Related papers  (2024-11-20T17:22:31Z)
• Conformer-based End-to-end Speech Recognition With Rotary Position Embedding [11.428057887454008]
  We introduce rotary position embedding (RoPE) in the convolution-augmented transformer (conformer) RoPE encodes absolute positional information into the input sequence by a rotation matrix, and then naturally incorporates explicit relative position information into a self-attention module. Our model achieves a relative word error rate reduction of 8.70% and 7.27% over the conformer on test-clean and test-other sets of the LibriSpeech corpus respectively.
  arXiv  Detail & Related papers  (2021-07-13T08:07:22Z)
• RoFormer: Enhanced Transformer with Rotary Position Embedding [9.01819510933327]
  We propose a novel method named Rotary Position Embedding(RoPE) to effectively leverage the positional information. RoPE encodes the absolute position with a rotation matrix and meanwhile incorporates the explicit relative position dependency in self-attention formulation. We evaluate the enhanced transformer with rotary position embedding, also called RoFormer, on various long text classification benchmark datasets.
  arXiv  Detail & Related papers  (2021-04-20T09:54:06Z)
• Rethinking Positional Encoding in Language Pre-training [111.2320727291926]
  We show that in absolute positional encoding, the addition operation applied on positional embeddings and word embeddings brings mixed correlations. We propose a new positional encoding method called textbfTransformer with textbfUntied textPositional textbfEncoding (T)
  arXiv  Detail & Related papers  (2020-06-28T13:11:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.