EndPrompt: Efficient Long-Context Extension via Terminal Anchoring

• URL: http://arxiv.org/abs/2605.14589v1
• Date: Thu, 14 May 2026 09:00:03 GMT
• Title: EndPrompt: Efficient Long-Context Extension via Terminal Anchoring
• Abstract summary: We propose EndPrompt, a method that achieves effective context extension using only short training sequences.<n>We preserve the original short context as an intact first segment and append a brief terminal prompt as a second segment, assigning it positional indices near the target context length.<n>EndPrompt achieves an average RULER score of 76.03 and the highest average on LongBench, surpassing LCEG (72.24), LongLoRA (72.95), and full-length fine-tuning.
• Score: 62.81677226065374
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Extending the context window of large language models typically requires training on sequences at the target length, incurring quadratic memory and computational costs that make long-context adaptation expensive and difficult to reproduce. We propose EndPrompt, a method that achieves effective context extension using only short training sequences. The core insight is that exposing a model to long-range relative positional distances does not require constructing full-length inputs: we preserve the original short context as an intact first segment and append a brief terminal prompt as a second segment, assigning it positional indices near the target context length. This two-segment construction introduces both local and long-range relative distances within a short physical sequence while maintaining the semantic continuity of the training text--a property absent in chunk-based simulation approaches that split contiguous context. We provide a theoretical analysis grounded in Rotary Position Embedding and the Bernstein inequality, showing that position interpolation induces a rigorous smoothness constraint over the attention function, with shared Transformer parameters further suppressing unstable extrapolation to unobserved intermediate distances. Applied to LLaMA-family models extending the context window from 8K to 64K, EndPrompt achieves an average RULER score of 76.03 and the highest average on LongBench, surpassing LCEG (72.24), LongLoRA (72.95), and full-length fine-tuning (69.23) while requiring substantially less computation. These results demonstrate that long-context generalization can be induced from sparse positional supervision, challenging the prevailing assumption that dense long-sequence training is necessary for reliable context-window extension. The code is available at https://github.com/clx1415926/EndPrompt.

Related papers

• Shuffle the Context: RoPE-Perturbed Self-Distillation for Long-Context Adaptation [86.62036852878354]
  Large language models (LLMs) increasingly operate in settings that require reliable long-context understanding.<n>We propose RoPE-Perturbed Self-Distillation, a training regularizer that improves positional robustness.<n> Experiments on long-context adaptation of Llama-3-8B and Qwen-3-4B demonstrate consistent gains on long-context benchmarks.
  arXiv  Detail & Related papers  (2026-04-15T18:46:35Z)
• Towards Infinite Length Extrapolation: A Unified Approach [0.0]
  Large language models (LLMs) have revolutionized natural language processing, but their ability to process long sequences is fundamentally limited by the context window size during training.<n>We use a unified framework that reinterprets positional encoding methods as a decomposition of the attention score into a multiplicative transformation and an additive bias.<n>Our theoretical analysis establishes conditions for infinite-context extrapolation, ensuring that the softmax handling remains well-defined over unbounded sequences while preserving long-distance correlations, entropy boundedness and gradient positional sensitivity.
  arXiv  Detail & Related papers  (2026-01-03T14:10:23Z)
• $π$-Attention: Periodic Sparse Transformers for Efficient Long-Context Modeling [5.216774377033164]
  PiAttention is a periodic sparse Transformer that factorizes attention into ring-local neighborhoods.<n>PiAttention matches or surpasses dense attention quality with 8.3% lower perplexity than RingAttention.
  arXiv  Detail & Related papers  (2025-11-12T09:09:13Z)
• Beyond Length: Quantifying Long-Range Information for Long-Context LLM Pretraining Data [67.46386646195818]
  We introduce LongFilter, a framework for curating training data tailored to long-context pretraining.<n>LongFilter measures the information gain provided by extended context by contrasting model predictions under long-context versus short-context settings.<n>Experiments with LLaMA-3-8B, extending its context length from 8K to 64K, show that LongFilter efficiently selects high-quality data and yields substantial improvements on benchmarks such as HELMET, LongBench, and RULER.
  arXiv  Detail & Related papers  (2025-10-29T06:21:08Z)
• LaMPE: Length-aware Multi-grained Positional Encoding for Adaptive Long-context Scaling Without Training [45.74983991122073]
  Large language models (LLMs) experience significant performance degradation when the input exceeds the pretraining context window.<n>Recent studies mitigate this problem by remapping OOD positions into the in-distribution range with fixed mapping strategies.<n>We propose Length-aware Multi-grained Positional Scaling (LaMPE), a training-free method that fully utilizes the model's effective context window.
  arXiv  Detail & Related papers  (2025-08-04T11:22:13Z)
• Long Context Alignment with Short Instructions and Synthesized Positions [56.1267385315404]
  This paper introduces Step-Skipping Alignment (SkipAlign) It is a new technique designed to enhance the long-context capabilities of Large Language Models (LLMs) With a careful selection of the base model and alignment datasets, SkipAlign with only 6B parameters achieves it's best performance and comparable with strong baselines like GPT-3.5-Turbo-16K on LongBench.
  arXiv  Detail & Related papers  (2024-05-07T01:56:22Z)
• CLEX: Continuous Length Extrapolation for Large Language Models [68.43814043853347]
  We propose Continuous Length EXtrapolation (CLEX) for Large Language Models (LLMs) CLEX extends the context window to over 4x or almost 8x training length, with no deterioration in performance. Our model trained on a 4k length exhibits competitive performance against state-of-the-art open-source models trained on context lengths up to 32k.
  arXiv  Detail & Related papers  (2023-10-25T08:13:02Z)
• LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models [67.58275666573496]
  LongLoRA is an efficient fine-tuning approach that extends the context sizes of pre-trained large language models. We demonstrate strong empirical results on various tasks on Llama2 models from 7B/13B to 70B.
  arXiv  Detail & Related papers  (2023-09-21T17:59:11Z)
• Extending Context Window of Large Language Models via Positional Interpolation [26.076599895589098]
  We present Position Interpolation that extends the context window sizes of RoPE-based pretrained LLMs to up to 32768 with minimal fine-tuning (within 1000 steps) We demonstrate strong empirical results on various tasks that require long context, including passkey retrieval, language modeling, and long document summarization from LLaMA 7B to 65B.
  arXiv  Detail & Related papers  (2023-06-27T16:26:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.