Shuffle the Context: RoPE-Perturbed Self-Distillation for Long-Context Adaptation

• URL: http://arxiv.org/abs/2604.14339v1
• Date: Wed, 15 Apr 2026 18:46:35 GMT
• Title: Shuffle the Context: RoPE-Perturbed Self-Distillation for Long-Context Adaptation
• Abstract summary: Large language models (LLMs) increasingly operate in settings that require reliable long-context understanding.<n>We propose RoPE-Perturbed Self-Distillation, a training regularizer that improves positional robustness.<n> Experiments on long-context adaptation of Llama-3-8B and Qwen-3-4B demonstrate consistent gains on long-context benchmarks.
• Score: 86.62036852878354
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) increasingly operate in settings that require reliable long-context understanding, such as retrieval-augmented generation and multi-document reasoning. A common strategy is to fine-tune pretrained short-context models at the target sequence length. However, we find that standard long-context adaptation can remain brittle: model accuracy depends strongly on the absolute placement of relevant evidence, exhibiting high positional variance even when controlling for task format and difficulty. We propose RoPE-Perturbed Self-Distillation, a training regularizer that improves positional robustness. The core idea is to form alternative "views" of the same training sequence by perturbing its RoPE indices -- effectively moving parts of the context to different positions -- and to train the model to produce consistent predictions across views via self-distillation. This encourages reliance on semantic signals instead of brittle position dependencies. Experiments on long-context adaptation of Llama-3-8B and Qwen-3-4B demonstrate consistent gains on long-context benchmarks, including up to 12.04% improvement on RULER-64K for Llama-3-8B and 2.71% on RULER-256K for Qwen-3-4B after SFT, alongside improved length extrapolation beyond the training context window.

Related papers

• REER-PT: Reverse-Engineered Reasoning for Perplexity-Guided Pre-training Data Augmentation [64.84437332310516]
  We introduce textbfREER-PT, a scalable framework that extends Reverse-Engineered Reasoning (REER) to raw pre-training data.<n>REER-PT identifies continuations that are difficult to predict but can still be inferred from the preceding context, and inserts concise reasoning annotations that reconstruct the missing connection between context and continuation.<n>This sparse transformation preserves the source text and remains compatible with standard next-token prediction, avoiding online reasoning rollouts during pre-training.
  arXiv  Detail & Related papers  (2026-08-31T11:34:19Z)
• Find, Fix, Reason: Context Repair for Video Reasoning [45.021693494492666]
  Reinforcement learning has advanced video reasoning in large multi-modal models.<n>We introduce an observation-level intervention: a frozen, tool-integrated teacher identifies the missingtemporal dependency and provides a minimal evidence patch.<n>We propose a Robust Improvement Reward (RIR) that aligns optimization with two goals: outcome validity through correct answers and dependency alignment rationales.
  arXiv  Detail & Related papers  (2026-04-17T17:04:19Z)
• PolicyLong: Towards On-Policy Context Extension [23.33185151859789]
  We propose PolicyLong, shifting data construction towards a dynamic on-policy paradigm.<n>By iteratively re-executing data screening, PolicyLong ensures the training distribution tracks evolving capabilities.<n>Experiments show PolicyLong consistently outperforms EntropyLong and NExtLong.
  arXiv  Detail & Related papers  (2026-04-09T05:07:57Z)
• SPELL: Self-Play Reinforcement Learning for evolving Long-Context Language Models [79.01078135582127]
  SPELL enables scalable, label-free optimization for long-context reasoning.<n>We introduce an automated curriculum that gradually increases document length and a reward function that adapts question difficulty to the model's evolving capabilities.
  arXiv  Detail & Related papers  (2025-09-28T13:08:10Z)
• Effective Length Extrapolation via Dimension-Wise Positional Embeddings Manipulation [60.22622442950905]
  Large Language Models (LLMs) often struggle to process and generate coherent context when the number of input tokens exceeds the pre-trained length.<n>Recent advancements in long-context extension have significantly expanded the context window of LLMs.<n>We propose Dimension-Wise Positional Embeddings Manipulation (DPE) to extrapolate the context window of LLMs.
  arXiv  Detail & Related papers  (2025-04-26T08:46:10Z)
• LongRoPE2: Near-Lossless LLM Context Window Scaling [46.936900701411965]
  LongRoPE2 is a novel approach that extends the effective context window of pre-trained large language models (LLMs) to the target length.<n>This is achieved by three contributions: (1) a hypothesis that insufficient training in higher RoPE dimensions contributes to the persistent out-of-distribution issues observed in existing methods; (2) an effective RoPE rescaling algorithm that adopts evolutionary search guided by "needle-driven" perplexity to address the insufficient training problem; and (3) a mixed context window training approach that fine-tunes model weights to adopt rescaled RoPE for long-context sequences.
  arXiv  Detail & Related papers  (2025-02-27T13:41:07Z)
• How to Train Long-Context Language Models (Effectively) [75.5418485597276]
  We study continued training and supervised fine-tuning (SFT) of a language model (LM) to make effective use of long-context information.<n>We find that code repositories and books are excellent sources of long data, but it is crucial to combine them with high-quality short-context data.<n>Our final model, ProLong-8B, demonstrates state-of-the-art long-context performance among similarly sized models at a length of 128K.
  arXiv  Detail & Related papers  (2024-10-03T16:46:52Z)
• Effective Long-Context Scaling of Foundation Models [90.57254298730923]
  We present a series of long-context LLMs that support effective context windows of up to 32,768 tokens. Our models achieve consistent improvements on most regular tasks and significant improvements on long-context tasks over Llama 2.
  arXiv  Detail & Related papers  (2023-09-27T21:41:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.