OPSDL: On-Policy Self-Distillation for Long-Context Language Models

• URL: http://arxiv.org/abs/2604.17535v1
• Date: Sun, 19 Apr 2026 16:53:56 GMT
• Title: OPSDL: On-Policy Self-Distillation for Long-Context Language Models
• Abstract summary: OPSDL is an On-Policy Self-Distillation method for enhancing the Long-context capabilities of large language models.<n>We evaluate OPSDL on long-context benchmarks across a range of models from 7B to 32B parameters.
• Score: 3.2617036218058413
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Extending the effective context length of large language models (LLMs) remains a central challenge for real-world applications. While recent post-training methods have made progress in long-context scaling, they either rely on high-quality supervision data or sparse sequence-level rewards, leading to unstable and inefficient optimization. We propose OPSDL, an On-Policy Self-Distillation method for enhancing the Long-context capabilities of LLMs. Unlike other recent self-distillation methods that inject privileged information and rely on the model's in-context learning ability to act as a teacher, OPSDL leverages the model's own inherently strong short-context capability as a self-teacher to supervise its own generation in long-context scenarios. The model first generates responses conditioned on the full long-context, then the self-teacher provides per-token supervision signals via point-wise reverse KL divergence under the relevant extracted short-context. This dense token-level signal encourages faithful use of relevant evidence and mitigates hallucinations induced by irrelevant context. We evaluate OPSDL on long-context benchmarks across a range of models from 7B to 32B parameters. Results show consistent and substantial improvements across varying context lengths, outperforming standard post-training approaches such as SFT and DPO with higher sample efficiency. Notably, these gains are achieved without degrading general short-context performance. These findings highlight the effectiveness of OPSDL as a scalable and stable approach for long-context learning.

Related papers

• Combining On-Policy Optimization and Distillation for Long-Context Reasoning in Large Language Models [21.053554352318738]
  We propose Distilled Group Relative Policy Optimization (dGRPO) for long-context reasoning.<n>We also introduce LongBlocks, a synthetic long-context dataset spanning multi-hop reasoning, contextual grounding, and long-form generation.<n>Our results show that combining outcome-based policy optimization with knowledge distillation in a single objective provides a more stable and effective path to long-context reasoning.
  arXiv  Detail & Related papers  (2026-05-12T15:04:18Z)
• Large Language Models for Sequential Decision-Making: Improving In-Context Learning via Supervised Fine-Tuning [0.7340845393655052]
  In this paper, we study the ICL capabilities of large language models (LLMs) in sequential decision-making settings.<n>We fine-tune pretrained LLMs to perform few-shot decision-making directly from offline, oracle-labeled trajectories.<n>Our framework enables flexible imitation of policies through supervised fine-tuning.
  arXiv  Detail & Related papers  (2026-05-09T15:49:49Z)
• Embedding-based In-Context Prompt Training for Enhancing LLMs as Text Encoders [20.530146256745414]
  EPIC is a novel embedding-based in-context prompt training strategy for large language models.<n>It generates high-quality embeddings while reducing computational burden during both training and inference.
  arXiv  Detail & Related papers  (2026-05-02T10:31:01Z)
• LongRM: Revealing and Unlocking the Context Boundary of Reward Modeling [45.520815757751194]
  We introduce Long-RewardBench, a benchmark specifically designed for long-context RM evaluation.<n>Preliminary study reveals that even state-of-the-art generative RMs exhibit significant fragility in long-context scenarios.<n>We propose a general multi-stage training strategy that effectively scales arbitrary models into robust Long-context RMs.
  arXiv  Detail & Related papers  (2025-10-08T11:48:16Z)
• Beyond In-Context Learning: Aligning Long-form Generation of Large Language Models via Task-Inherent Attribute Guidelines [71.14354526117958]
  In-context learning (ICL) is an important yet not fully understood ability of pre-trained large language models (LLMs)<n>We present LongGuide, which efficiently generates two parallel streams of guidelines capturing task language and format properties.<n>LongGuide automatically selects the best combination of guidelines, improving both strong open- and closed-source LLMs by over 5% in both zero- and few-shot settings.
  arXiv  Detail & Related papers  (2025-06-02T02:35:24Z)
• Disentangling Length Bias In Preference Learning Via Response-Conditioned Modeling [87.17041933863041]
  Reinforcement Learning from Human Feedback (RLHF) has achieved considerable success in aligning large language models (LLMs)<n>We introduce a $textbfR$esponse-$textbfc$onditioned $textbfB$radley-$textbfT$erry (Rc-BT) model that enhances the model's capability in length bias mitigating and length instruction following.<n>We also propose the Rc-RM and Rc-DPO algorithm to leverage the Rc-BT model for reward modeling and direct policy optimization
  arXiv  Detail & Related papers  (2025-02-02T14:50:25Z)
• What is Wrong with Perplexity for Long-context Language Modeling? [71.34933096461124]
  Long-context inputs are crucial for large language models (LLMs) in tasks such as extended conversations, document summarization, and many-shot in-context learning.<n>Perplexity (PPL) has proven unreliable for assessing long-context capabilities.<n>We propose bfLongPPL, a novel metric that focuses on key tokens by employing a long-short context contrastive method to identify them.
  arXiv  Detail & Related papers  (2024-10-31T09:39:28Z)
• LongReward: Improving Long-context Large Language Models with AI Feedback [54.3321542678909]
  LongReward is a novel method that provides rewards for long-context model responses from four human-valued dimensions. Our experiments indicate that LongReward not only significantly improves models' long-context performance but also enhances their ability to follow short instructions.
  arXiv  Detail & Related papers  (2024-10-28T17:50:42Z)
• GATEAU: Selecting Influential Samples for Long Context Alignment [59.579128690086385]
  GATEAU identifies influential samples enriched with long-range dependency relations.<n>Model trained on selected samples exhibits better instruction-following and long-context understanding capabilities.
  arXiv  Detail & Related papers  (2024-10-21T04:30:53Z)
• A Controlled Study on Long Context Extension and Generalization in LLMs [85.4758128256142]
  Broad textual understanding and in-context learning require language models that utilize full document contexts. Due to the implementation challenges associated with directly training long-context models, many methods have been proposed for extending models to handle long contexts. We implement a controlled protocol for extension methods with a standardized evaluation, utilizing consistent base models and extension data.
  arXiv  Detail & Related papers  (2024-09-18T17:53:17Z)
• LongSkywork: A Training Recipe for Efficiently Extending Context Length in Large Language Models [61.12177317970258]
  LongSkywork is a long-context Large Language Model capable of processing up to 200,000 tokens. We develop two novel methods for creating synthetic data. LongSkywork achieves outstanding performance on a variety of long-context benchmarks.
  arXiv  Detail & Related papers  (2024-06-02T03:34:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.