EasyOPD: An Easy-to-use On-Policy Distillation Framework for Large Language Models

• URL: http://arxiv.org/abs/2607.11012v1
• Date: Mon, 13 Jul 2026 02:27:50 GMT
• Title: EasyOPD: An Easy-to-use On-Policy Distillation Framework for Large Language Models
• Abstract summary: We present textscEasyOPD, an on-policy distillation framework built on verl.<n>textscEasyOPD separates user-side configuration, method-specific supervision logic, and verl-based execution.<n>We instantiate representative methods for three OPD settings -- cross-tokenizer OPD, on-policy self-distillation, and step-wise OPD.
• Score: 39.18361000090602
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Conventional language-model distillation often relies on fixed teacher-generated data, which may not cover the states encountered by an evolving student policy. On-policy distillation (OPD) instead collects teacher or evaluator supervision on student-generated rollouts. However, existing OPD methods differ substantially in supervision form, tokenizer compatibility, teacher access, and supervision granularity, leading to fragmented implementations that are difficult to reproduce and extend. We present \textsc{EasyOPD}, an on-policy distillation framework built on verl, a distributed reinforcement-learning framework for large language models. \textsc{EasyOPD} separates user-side configuration, method-specific supervision logic, and verl-based execution. Its method modules connect to the shared backend through extension boundaries for loss construction, rollout metadata, reward processing, tokenizer alignment, and teacher-side computation. We instantiate representative methods for three OPD settings -- cross-tokenizer OPD, on-policy self-distillation, and step-wise OPD. Experiments on reasoning, code-generation, scientific-knowledge, and tool-use benchmarks show that these implementations can be executed through the same verl-based backend while retaining their method-specific objectives and task-dependent performance profiles. We release \textsc{EasyOPD} with runnable YAML configurations, documentation, and an installable demonstration package and video.

Related papers

• DHRCL:Training Code LLMs with Dense Hierarchical Rewards and Curriculum Learning [15.421698703970167]
  We propose DHRCL, a reinforcement learning framework with Dense Hierarchical Rewards and Curriculum Learning.<n>DHRCL decomposes feedback into syntax validation, execution success, unit-test pass rate, and AST-based structural similarity.<n>We evaluate DHRCL across Qwen3-4B, Qwen3-8B, and Qwen3-14B backbones, showing that its advantage remains consistent as model capacity increases.
  arXiv  Detail & Related papers  (2026-07-29T04:16:33Z)
• dOPSD: On-Policy Self-Distillation for Diffusion Language Models [52.60302464420127]
  Diffusion large language models generate text by iteratively denoising a masked sequence.<n>Supervised fine-tuning is off-policy and suffers from exposure bias, while reinforcement learning gives only sparse, sequence-level rewards.<n>On-policy self-distillation (OPSD) offers a promising alternative, using one model as both student and teacher.
  arXiv  Detail & Related papers  (2026-07-05T17:47:29Z)
• Curriculum Learning-Guided Progressive Distillation in Large Language Models [18.961191510804134]
  Existing distillation approaches often overlook two critical factors: the learning order of training data and the capacity mismatch between teacher and student models.<n>We propose Curriculum Learning-Guided Progressive Distillation (CLPD), a unified framework that accounts for both factors by aligning data difficulty with teacher strength.<n>Our framework is modular and can be integrated into standard distillation algorithms with minimal overhead.
  arXiv  Detail & Related papers  (2026-05-11T21:37:20Z)
• Rubric-based On-policy Distillation [62.11106822527392]
  On-policy distillation (OPD) is a powerful paradigm for model alignment, yet its reliance on teacher logits restricts its application to white-box scenarios.<n>We contend that structured semantic rubrics can serve as a scalable alternative to teacher logits, enabling OPD using only teacher-generated responses.<n>We introduce ROPD, a simple yet foundational framework for rubric-based OPD. Specifically, ROPD induces prompt-specific rubrics from teacher-student contrasts, and then utilizes these rubrics to score the student rollouts for on-policy optimization.
  arXiv  Detail & Related papers  (2026-05-08T07:52:15Z)
• Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe [53.40076304466524]
  We propose Uni-OPD, a unified OPD framework that generalizes across Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs)<n>Specifically, from the student's perspective, we adopt two data balancing strategies to promote exploration of informative student-generated states during training.<n>We develop an outcome-guided margin calibration mechanism to restore order consistency between correct and incorrect trajectories.
  arXiv  Detail & Related papers  (2026-05-05T12:15:21Z)
• Data-Model Co-Evolution: Growing Test Sets to Refine LLM Behavior [10.041741229516141]
  Large Language Models (LLMs) allow developers to govern model behavior by editing prompt instructions.<n>We operationalize this paradigm in an interactive system designed to address the challenge of encoding subtle, domain-specific policies into prompt instructions.
  arXiv  Detail & Related papers  (2025-10-14T17:07:37Z)
• AgenticIE: An Adaptive Agent for Information Extraction from Complex Regulatory Documents [1.338174941551702]
  Declaration of Performance (DoP) documents, mandated by EU regulation, certify the performance of construction products.<n>There are two challenges to make DoPs machine and human accessible through automated key-value pair extraction (KVP) and question answering (QA)
  arXiv  Detail & Related papers  (2025-09-15T10:53:05Z)
• Integrated Structural Prompt Learning for Vision-Language Models [15.002501540565781]
  In this paper, we propose an Integrated Structural Prompt (ISP) for Vision-Language Models (VLMs)<n>ISP introduces self-structural and cross-structural prompt modules to model the structural relationships between learnable prompts and frozen tokens.<n>ISP achieves competitive performance against state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-07-08T04:59:58Z)
• Structured Agent Distillation for Large Language Model [56.38279355868093]
  We propose Structured Agent Distillation, a framework that compresses large LLM-based agents into smaller student models.<n>Our method segments trajectories into [REASON] and [ACT] spans, applying segment-specific losses to align each component with the teacher's behavior.<n>Experiments on ALFWorld, HotPotQA-ReAct, and WebShop show that our approach consistently outperforms token-level and imitation learning baselines.
  arXiv  Detail & Related papers  (2025-05-20T02:01:55Z)
• CELA: Cost-Efficient Language Model Alignment for CTR Prediction [70.65910069412944]
  Click-Through Rate (CTR) prediction holds a paramount position in recommender systems.<n>Recent efforts have sought to mitigate these challenges by integrating Pre-trained Language Models (PLMs)<n>We propose textbfCost-textbfEfficient textbfLanguage Model textbfAlignment (textbfCELA) for CTR prediction.
  arXiv  Detail & Related papers  (2024-05-17T07:43:25Z)
• Learning Multi-Objective Curricula for Deep Reinforcement Learning [55.27879754113767]
  Various automatic curriculum learning (ACL) methods have been proposed to improve the sample efficiency and final performance of deep reinforcement learning (DRL) In this paper, we propose a unified automatic curriculum learning framework to create multi-objective but coherent curricula. In addition to existing hand-designed curricula paradigms, we further design a flexible memory mechanism to learn an abstract curriculum.
  arXiv  Detail & Related papers  (2021-10-06T19:30:25Z)
• CINS: Comprehensive Instruction for Few-shot Learning in Task-oriented Dialog Systems [56.302581679816775]
  This paper proposes Comprehensive Instruction (CINS) that exploits PLMs with task-specific instructions. We design a schema (definition, constraint, prompt) of instructions and their customized realizations for three important downstream tasks in ToD. Experiments are conducted on these ToD tasks in realistic few-shot learning scenarios with small validation data.
  arXiv  Detail & Related papers  (2021-09-10T03:23:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.