Randomized YaRN Improves Length Generalization for Long-Context Reasoning

• URL: http://arxiv.org/abs/2606.23687v1
• Date: Mon, 22 Jun 2026 17:59:53 GMT
• Title: Randomized YaRN Improves Length Generalization for Long-Context Reasoning
• Abstract summary: We propose a training method that improves length generalization by combining YaRN-based positional extrapolation with randomized positional encoding and a length curriculum.<n>We evaluate Randomized YaRN on two challenging long-context reasoning benchmarks, BABILong and Multi-Round Coreference Resolution (MRCR)<n>When training on data with 8K context, Randomized YaRN consistently improves reasoning performance on context lengths from 16K to 128K and outperforms standard fine-tuning.
• Score: 38.265787556913175
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) are typically pretrained on short sequences and then extended to work on longer sequences with additional training. However, such LLMs still struggle to further generalize to very long sequences. We propose Randomized YaRN, a training method that improves length generalization by combining YaRN-based positional extrapolation with randomized positional encoding and a length curriculum. During training on short context data, tokens are assigned YaRN positional encodings sampled from a larger position range, exposing the model to out-of-distribution positional representations even on short-context inputs. We evaluate Randomized YaRN on two challenging long-context reasoning benchmarks, BABILong and Multi-Round Coreference Resolution (MRCR). When training on data with <8K context, Randomized YaRN consistently improves reasoning performance on context lengths from 16K to 128K and outperforms standard fine-tuning, with the largest gains appearing at far out-of-distribution lengths. Our results suggest that progressively exposing models to OOD positional distributions provides an effective recipe for generalizable long-context reasoning.

Related papers

• EndPrompt: Efficient Long-Context Extension via Terminal Anchoring [62.81677226065374]
  We propose EndPrompt, a method that achieves effective context extension using only short training sequences.<n>We preserve the original short context as an intact first segment and append a brief terminal prompt as a second segment, assigning it positional indices near the target context length.<n>EndPrompt achieves an average RULER score of 76.03 and the highest average on LongBench, surpassing LCEG (72.24), LongLoRA (72.95), and full-length fine-tuning.
  arXiv  Detail & Related papers  (2026-05-14T09:00:03Z)
• Shuffle the Context: RoPE-Perturbed Self-Distillation for Long-Context Adaptation [86.62036852878354]
  Large language models (LLMs) increasingly operate in settings that require reliable long-context understanding.<n>We propose RoPE-Perturbed Self-Distillation, a training regularizer that improves positional robustness.<n> Experiments on long-context adaptation of Llama-3-8B and Qwen-3-4B demonstrate consistent gains on long-context benchmarks.
  arXiv  Detail & Related papers  (2026-04-15T18:46:35Z)
• Position Encoding with Random Float Sampling Enhances Length Generalization of Transformers [27.92897019476987]
  We introduce a powerful position encoding (PE) strategy, Random Float Sampling (RFS), that generalizes well to lengths unseen during pretraining or fine-tuning.<n>RFS uses randomly sampled continuous values, thereby avoiding out-of-distribution (OOD) issues on unseen lengths by exposing the model to diverse indices during training.<n> Experiments corroborate its effectiveness by showing that RFS results in superior performance in length generalization tasks as well as zero-shot commonsense reasoning benchmarks.
  arXiv  Detail & Related papers  (2026-02-15T08:32:22Z)
• Short-Context Dominance: How Much Local Context Natural Language Actually Needs? [48.429870236229696]
  We measure the minimum context length needed to reproduce accurate full-context predictions.<n>For sequences with 1-7k tokens from long-context documents, we consistently find that 75-80% require only the last 96 tokens at most.<n>We introduce a practical proxy to MCL, called Distributionally Aware MCL (DaMCL), that does not require knowledge of the actual next-token.
  arXiv  Detail & Related papers  (2025-12-08T22:25:00Z)
• Chunks as Arms: Multi-Armed Bandit-Guided Sampling for Long-Context LLM Preference Optimization [56.97588709890706]
  LongMab-PO is a novel framework that generates high-quality and diverse responses for long-context modeling tasks.<n> Experimental results show that LongMab-PO significantly improves the diversity and quality of preference data pairs.
  arXiv  Detail & Related papers  (2025-08-19T16:33:55Z)
• Understanding and Improving Length Generalization in Recurrent Models [21.417068818034817]
  recurrent models can process arbitrarily long sequences, but their performance sometimes drops considerably beyond their training context lengths.<n>We show that models fail to length generalize when during training they are only exposed to a limited subset of the distribution of all attainable states.<n>We investigate simple training interventions that aim to increase the coverage of the states that the model is trained on.
  arXiv  Detail & Related papers  (2025-07-03T16:45:50Z)
• Position as Probability: Self-Supervised Transformers that Think Past Their Training for Length Extrapolation [0.0]
  PRISM is a novel positional encoding mechanism that enables Transformers to extrapolate accurately up to 10x beyond their training length.<n>Our analysis demonstrates that PRISM's positional encoding maintains sharp and interpretable internal states, providing a theoretical basis for reliable length generalization.
  arXiv  Detail & Related papers  (2025-06-01T09:20:44Z)
• LongAlign: A Recipe for Long Context Alignment of Large Language Models [61.85923382850057]
  LongAlign is a recipe of the instruction data, training, and evaluation for long context alignment. We construct a long instruction-following dataset using Self-Instruct. We adopt the packing and sorted strategies to speed up supervised fine-tuning on data with varied length distributions.
  arXiv  Detail & Related papers  (2024-01-31T18:29:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.