Distance generalization in transformers: why bother with positional encoding?

• URL: http://arxiv.org/abs/2609.11913v1
• Date: Thu, 10 Sep 2026 17:57:19 GMT
• Title: Distance generalization in transformers: why bother with positional encoding?
• Abstract summary: We focus on distance generalization, which probes performance when inter-token distances are changed between training and inference.<n>We construct two synthetic delay copy tasks, both involving finite distances between source and recall, and test models on delays unseen during training.
• Score: 1.7188280334580195
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Out-of-distribution length generalization, namely to extrapolate a task from short to longer context, has been studied intensively for transformers. Here we focus on distance generalization, which probes performance when inter-token distances are changed between training and inference, while keeping a fixed context length. We construct two synthetic delay copy tasks, both involving finite distances between source and recall, where tokens are copied either fully or selectively, and test models on delays unseen during training. We address three questions: (A) Do positional encoding schemes such as RoPE and ALiBi improve distance resolution relative to no positional encoding (NoPE)? (B) How does data diversity, the number of inter-token distances seen in training, affect performance? (C) When is distance transfer learning positive or negative? We present a thorough investigation, finding that it is paramount to improve our understanding of the underlying mechanisms.

Related papers

• Relative Positions Generalize, Absolute Positions Memorize: An Implicit-Bias Account of Length Generalization in Attention [4.422349568747053]
  Transformers with relative positional encodings often extrapolate to sequences longer than those seen during training.<n>We prove that rotary encodings make the attention logit a function of relative offset alone.<n> Learned absolute encodings instead leave out-of-range positions unconstrained, and the trained head pins to a fixed absolute position inside the training range.
  arXiv  Detail & Related papers  (2026-07-21T06:29:10Z)
• Learning the Koopman Operator using Attention Free Transformers [0.3848364262836075]
  We introduce two complementary components that make Koopman predictors more robust.<n>First, we add an attention-free latent memory (AFT) block that aggregates a short window of past latents to produce a corrected latent before each Koopman update.<n>Second, we propose dynamic re-encoding: lightweight, online change-point triggers that detect latent drift and project predictions back onto the autoencoder manifold.
  arXiv  Detail & Related papers  (2026-06-22T21:36:55Z)
• Improving Robotic Imitation Learning via Trajectory Standardization [7.969825103657315]
  A common preprocessing strategy is time-uniform downsampling to shorten sequences, but it cannot effectively remove speed-induced non-uniformity or redundant pauses.<n>We propose Information-Standardized Trajectory Resampling (ISR), an offline preprocessing method for effective imitation learning.<n>ISR improves task success rates by about 25%, remains robust across datasets collected from different operators, and reduces both dataset size and training cost.
  arXiv  Detail & Related papers  (2026-06-22T06:46:26Z)
• TAD: Temporal-Aware Trajectory Self-Distillation for Fast and Accurate Diffusion LLM [18.69783590125584]
  Diffusion large language models (dLLMs) offer a promising paradigm for parallel text generation.<n>They face an accuracy-parallelism trade-off, where increasing tokens per forward (TPF) often degrades generation quality.<n>Existing acceleration methods often gain speed at the cost of accuracy.<n>We propose TAD, a Temporal-Aware trajectory self-Distillation framework.
  arXiv  Detail & Related papers  (2026-05-10T13:38:53Z)
• How Transformers Learn Regular Language Recognition: A Theoretical Study on Training Dynamics and Implicit Bias [48.9399496805422]
  We focus on two representative tasks in the category of regular language recognition, known as even pairs' and parity check'<n>Our goal is to explore how a one-layer transformer, consisting of an attention layer followed by a linear layer, learns to solve these tasks.
  arXiv  Detail & Related papers  (2025-05-02T00:07:35Z)
• Context-aware Biases for Length Extrapolation [0.19116784879310025]
  We propose an additive RPE, Context-Aware Biases for Length Extrapolation (CABLE)<n>CABLE learns token-specific, context-aware biases for each attention head in transformers.<n>Our method significantly enhances the performance of existing RPE methods tested on the FineWeb-Edu-10B and WikiText-103 datasets.
  arXiv  Detail & Related papers  (2025-03-11T05:54:58Z)
• Transformers Can Achieve Length Generalization But Not Robustly [76.06308648699357]
  We show that the success of length generalization is intricately linked to the data format and the type of position encoding. We show for the first time that standard Transformers can extrapolate to a sequence length that is 2.5x the input length.
  arXiv  Detail & Related papers  (2024-02-14T18:18:29Z)
• A Length-Extrapolatable Transformer [98.54835576985664]
  We focus on length extrapolation, i.e., training on short texts while evaluating longer sequences. We introduce a relative position embedding to explicitly maximize attention resolution. We evaluate different Transformer variants with language modeling.
  arXiv  Detail & Related papers  (2022-12-20T18:56:20Z)
• Dissecting Transformer Length Extrapolation via the Lens of Receptive Field Analysis [72.71398034617607]
  We dissect a relative positional embedding design, ALiBi, via the lens of receptive field analysis. We modify the vanilla Sinusoidal positional embedding to create bftext, the first parameter-free relative positional embedding design that truly length information uses longer than the training sequence.
  arXiv  Detail & Related papers  (2022-12-20T15:40:17Z)
• Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search [84.94597821711808]
  We extend PoWER-BERT (Goyal et al., 2020) and propose Length-Adaptive Transformer that can be used for various inference scenarios after one-shot training. We conduct a multi-objective evolutionary search to find a length configuration that maximizes the accuracy and minimizes the efficiency metric under any given computational budget. We empirically verify the utility of the proposed approach by demonstrating the superior accuracy-efficiency trade-off under various setups.
  arXiv  Detail & Related papers  (2020-10-14T12:28:08Z)
• DA-Transformer: Distance-aware Transformer [87.20061062572391]
  DA-Transformer is a distance-aware Transformer that can exploit the real distance. In this paper, we propose DA-Transformer, which is a distance-aware Transformer that can exploit the real distance.
  arXiv  Detail & Related papers  (2020-10-14T10:09:01Z)
• Learning Invariant Representations for Reinforcement Learning without Reconstruction [98.33235415273562]
  We study how representation learning can accelerate reinforcement learning from rich observations, such as images, without relying either on domain knowledge or pixel-reconstruction. Bisimulation metrics quantify behavioral similarity between states in continuous MDPs. We demonstrate the effectiveness of our method at disregarding task-irrelevant information using modified visual MuJoCo tasks.
  arXiv  Detail & Related papers  (2020-06-18T17:59:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.