Position Encoding with Random Float Sampling Enhances Length Generalization of Transformers

• URL: http://arxiv.org/abs/2602.14050v1
• Date: Sun, 15 Feb 2026 08:32:22 GMT
• Title: Position Encoding with Random Float Sampling Enhances Length Generalization of Transformers
• Abstract summary: We introduce a powerful position encoding (PE) strategy, Random Float Sampling (RFS), that generalizes well to lengths unseen during pretraining or fine-tuning.<n>RFS uses randomly sampled continuous values, thereby avoiding out-of-distribution (OOD) issues on unseen lengths by exposing the model to diverse indices during training.<n> Experiments corroborate its effectiveness by showing that RFS results in superior performance in length generalization tasks as well as zero-shot commonsense reasoning benchmarks.
• Score: 27.92897019476987
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Length generalization is the ability of language models to maintain performance on inputs longer than those seen during pretraining. In this work, we introduce a simple yet powerful position encoding (PE) strategy, Random Float Sampling (RFS), that generalizes well to lengths unseen during pretraining or fine-tuning. In particular, instead of selecting position indices from a predefined discrete set, RFS uses randomly sampled continuous values, thereby avoiding out-of-distribution (OOD) issues on unseen lengths by exposing the model to diverse indices during training. Since assigning indices to tokens is a common and fundamental procedure in widely used PEs, the advantage of RFS can easily be incorporated into, for instance, the absolute sinusoidal encoding, RoPE, and ALiBi. Experiments corroborate its effectiveness by showing that RFS results in superior performance in length generalization tasks as well as zero-shot commonsense reasoning benchmarks.

Related papers

• Stabilizing Extrapolation in Looped Transformers via Learned Stochastic Stopping [56.14767235650558]
  Looped Transformers, which repeatedly apply a shared transformer block, are an architecturally natural fit for variable-length computation tasks.<n>We trace this variance to the spurious correlation in simple algorithmic tasks between sequence length and number of loops.<n>Our work suggests that "when to stop" should be treated as a training design choice, not merely an inference-time-allocation rule.
  arXiv  Detail & Related papers  (2026-06-29T08:58:09Z)
• Randomized YaRN Improves Length Generalization for Long-Context Reasoning [38.265787556913175]
  We propose a training method that improves length generalization by combining YaRN-based positional extrapolation with randomized positional encoding and a length curriculum.<n>We evaluate Randomized YaRN on two challenging long-context reasoning benchmarks, BABILong and Multi-Round Coreference Resolution (MRCR)<n>When training on data with 8K context, Randomized YaRN consistently improves reasoning performance on context lengths from 16K to 128K and outperforms standard fine-tuning.
  arXiv  Detail & Related papers  (2026-06-22T17:59:53Z)
• Self-Supervised Learning via Flow-Guided Neural Operator on Time-Series Data [57.85958428020496]
  Flow-Guided Neural Operator (FGNO) is a novel framework combining operator learning with flow matching for SSL training.<n>FGNO learns mappings in functional spaces by using Short-Time Fourier Transform to unify different time resolutions.<n>Unlike prior generative SSL methods that use noisy inputs during inference, we propose using clean inputs for representation extraction while learning representations with noise.
  arXiv  Detail & Related papers  (2026-02-12T18:54:57Z)
• Predicting LLM Output Length via Entropy-Guided Representations [13.351384070796747]
  We introduce a lightweight framework that reuses the main model's internal hidden states for efficient length prediction.<n>Our framework features two core components: 1) Entropy-Guided Token Pooling (EGTP), which uses on-the-fly activations and token entropy for highly accurate static prediction.
  arXiv  Detail & Related papers  (2026-02-12T10:49:04Z)
• From Shortcut to Induction Head: How Data Diversity Shapes Algorithm Selection in Transformers [67.02076505996284]
  We study how the choice of pretraining data distribution steers a shallow transformer toward one behavior or the other.<n>Our results shed light on the algorithmic biases of pretrained transformers and offer conceptual guidelines for data-driven control of their learned behaviors.
  arXiv  Detail & Related papers  (2025-12-21T08:10:26Z)
• Wavelet-based Disentangled Adaptive Normalization for Non-stationary Times Series Forecasting [23.34966767653385]
  We propose Wavelet-based Disentangled Adaptive Normalization (WDAN) to address non-stationarity in time series forecasting.<n>WDAN uses discrete wavelet transforms to break down the input into low-frequency trends and high-frequency fluctuations.<n>Experiments on multiple benchmarks demonstrate that WDAN consistently improves forecasting accuracy across various backbone model.
  arXiv  Detail & Related papers  (2025-06-06T08:25:29Z)
• Position as Probability: Self-Supervised Transformers that Think Past Their Training for Length Extrapolation [0.0]
  PRISM is a novel positional encoding mechanism that enables Transformers to extrapolate accurately up to 10x beyond their training length.<n>Our analysis demonstrates that PRISM's positional encoding maintains sharp and interpretable internal states, providing a theoretical basis for reliable length generalization.
  arXiv  Detail & Related papers  (2025-06-01T09:20:44Z)
• Context-aware Biases for Length Extrapolation [0.19116784879310025]
  We propose an additive RPE, Context-Aware Biases for Length Extrapolation (CABLE)<n>CABLE learns token-specific, context-aware biases for each attention head in transformers.<n>Our method significantly enhances the performance of existing RPE methods tested on the FineWeb-Edu-10B and WikiText-103 datasets.
  arXiv  Detail & Related papers  (2025-03-11T05:54:58Z)
• Adapting to Length Shift: FlexiLength Network for Trajectory Prediction [53.637837706712794]
  Trajectory prediction plays an important role in various applications, including autonomous driving, robotics, and scene understanding. Existing approaches mainly focus on developing compact neural networks to increase prediction precision on public datasets, typically employing a standardized input duration. We introduce a general and effective framework, the FlexiLength Network (FLN), to enhance the robustness of existing trajectory prediction against varying observation periods.
  arXiv  Detail & Related papers  (2024-03-31T17:18:57Z)
• On Optimal Sampling for Learning SDF Using MLPs Equipped with Positional Encoding [76.78321723602636]
  We devise a tool to determine the appropriate sampling rate for learning an accurate neural implicit field without undesirable side effects.<n>It is observed that a PE-equipped has an intrinsic frequency much higher than the highest frequency component in the PE layer.<n>We empirically show in the setting of SDF fitting, this recommended sampling rate is sufficient to secure accurate fitting results.
  arXiv  Detail & Related papers  (2024-01-02T10:51:52Z)
• On Convergence of Incremental Gradient for Non-Convex Smooth Functions [63.51187646914962]
  In machine learning and network optimization, algorithms like shuffle SGD are popular due to minimizing the number of misses and good cache. This paper delves into the convergence properties SGD algorithms with arbitrary data ordering.
  arXiv  Detail & Related papers  (2023-05-30T17:47:27Z)
• Random Feature Attention [69.4671822971207]
  We propose RFA, a linear time and space attention that uses random feature methods to approximate the softmax function. RFA can be used as a drop-in replacement for conventional softmax attention and offers a straightforward way of learning with recency bias through an optional gating mechanism. Experiments on language modeling and machine translation demonstrate that RFA achieves similar or better performance compared to strong transformer baselines.
  arXiv  Detail & Related papers  (2021-03-03T02:48:56Z)
• FASA: Feature Augmentation and Sampling Adaptation for Long-Tailed Instance Segmentation [91.129039760095]
  Recent methods for long-tailed instance segmentation still struggle on rare object classes with few training data. We propose a simple yet effective method, Feature Augmentation and Sampling Adaptation (FASA) FASA is a fast, generic method that can be easily plugged into standard or long-tailed segmentation frameworks.
  arXiv  Detail & Related papers  (2021-02-25T14:07:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.