On-Policy Distillation Meets Off-Policy GRPO: Training Compact Instruction-Following Rerankers

• URL: http://arxiv.org/abs/2609.01947v1
• Date: Tue, 01 Sep 2026 23:36:19 GMT
• Title: On-Policy Distillation Meets Off-Policy GRPO: Training Compact Instruction-Following Rerankers
• Abstract summary: We revisit reranker distillation through the lens of reinforcement learning.<n>We propose a two-stage framework combining off-policy teacher optimization with on-policy student distillation.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Compact instruction-following rerankers are attractive for deployment, but conventional distillation pipelines typically train students by offline imitation of teacher outputs on a fixed set of examples, constraining supervision to the teacher's observed ranking space. We revisit reranker distillation through the lens of reinforcement learning. We propose a two-stage framework combining off-policy teacher optimization with on-policy student distillation. In Stage 1, a 4B teacher reranker is strengthened with off-policy GRPO using LLM-judge feedback on 88K instruction-following examples. In Stage 2, a compact 1B student samples rankings from its own policy and receives soft teacher-derived rewards on those rankings, coupling student exploration with knowledge transfer. Our strongest gains appear under distribution shift. On MAIR-11, the original 11-subset, 869-query evaluation, the proposed student reaches 0.7670 nDCG@6, outperforming offline listwise KD by +4.6 points. Controlled comparisons against offline pairwise RankNet KD and on-policy GKD show that neither changing the offline distillation objective nor moving teacher-distribution matching on-policy reproduces the performance of reward-based on-policy distillation over student-sampled rankings. The advantage persists on MAIR-Full: across all 126 tasks and 9,356 queries, the proposed method obtains the highest task-macro point estimates among the evaluated distillation variants, reaching 0.6808 nDCG@6 and 0.7865 MRR@6. It also exceeds two released 7B RL-trained rerankers on the comparable MAIR-11 evaluation, while the same Stage 2 training procedure consistently improves three architecturally distinct alternative student backbones. On the 9,861-query validation benchmark, the resulting 1B reranker achieves 0.7624 nDCG@6 while providing a favorable quality-efficiency tradeoff relative to larger alternatives.

Related papers

• Eliciting Weak-to-Strong Generalization with On-Policy Reverse Distillation [41.854144563456046]
  Weak-to-strong generalization asks whether stronger models can learn from weaker supervisors and surpass them.<n>We introduce On-Policy Reverse Distillation, which evaluates the teacher's policy shift and amplifies the student's verifier-driven policy gradient.<n>In both successive model transfer and multi-teacher distillation, OPRD achieves prohibitively higher performance with fewer student updates than existing RL and distillation approaches.
  arXiv  Detail & Related papers  (2026-09-08T14:26:35Z)
• CORE: Improving Compositional Reasoning in MLLM Embedding via Reranker Distillation [49.0564416580726]
  We distill the compositional judgments of a cross-attentive reranker into an embedding model.<n>We show that CORE-RERANKER-8B achieves an 82.7% total average, outperforming Jina-Reranker by 10.7 points.<n>The improvements transfer to the MCMR benchmark without sacrificing retrieval performance on COCO and Flickr30K.
  arXiv  Detail & Related papers  (2026-09-03T16:50:29Z)
• Distill Skills into Weights, Not Prompts: Abstract Skills as Privileged Signals for On-Policy Self-Distillation [7.296659715681798]
  Reinforcement learning with verifiable rewards yields no group-relative signal when rollout groups are uniformly correct or uniformly wrong.<n>We propose SKALD (Skill-Anchored Latent Distillation), an on-policy self-distillation framework that uses two context views of the same Qwen3-Base model.
  arXiv  Detail & Related papers  (2026-08-10T16:43:18Z)
• Pass the Baton: Trajectory-Relayed On-Policy Distillation [23.50689651536186]
  On-policy distillation grounds token-level supervision in the student's own trajectory, yet suffers from prefix failure.<n>We identify a teacher-student continuation asymmetry on failed prefixes, and convert it into a label-free handoff trigger in Relay On-Policy Distillation (Relay-OPD)<n>During training, Relay-OPD constructs relay trajectories by letting the teacher briefly take over at detected trigger points to produce a teacher leg, after which the student resumes and is optimized on the resulting trajectory.
  arXiv  Detail & Related papers  (2026-07-28T17:59:46Z)
• Beyond Scalar Rewards by Internalizing Reasoning into Score Distributions [80.33276868889341]
  Z-Reward decouples reasoning-heavy judgment from efficient reward deployment.<n>We show that Z-Reward can serve as a differentiable reward signal for text-to-image optimization.
  arXiv  Detail & Related papers  (2026-06-08T06:20:12Z)
• When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoning [45.79647925282674]
  On-policy self-distillation (OPSD) trains a student on its own rollouts using a privileged teacher.<n>Existing entropy-based OPD methods relax this uniformity by modulating token-level supervision with teacher entropy.<n>We propose Position-Weighted On-Policy Self-Distillation (PW-OPSD), which applies an increasing position weight while keeping the same student rollout, privileged teacher pass, and clipped forward-KL target as OPSD.
  arXiv  Detail & Related papers  (2026-05-20T18:14:03Z)
• Teacher-Guided Policy Optimization for LLM Distillation [90.49982387646861]
  Teacher-Guided Policy Optimization (TGPO) is an on-policy algorithm that incorporates dense directional guidance by leveraging teacher predictions conditioned on the student's rollout.<n>Experiments on complex reasoning benchmarks demonstrate that TGPO significantly outperforms standard baselines and is robust to different teachers.
  arXiv  Detail & Related papers  (2026-05-13T09:20:03Z)
• On-Policy Distillation with Best-of-N Teacher Rollout Selection [54.91780727674628]
  We propose BRTS, a Best-of-N Rollout Teacher Selection framework for on-policy distillation.<n>BRTS augments standard student-context OPD with a teacher-context supervision branch constructed from the teacher trajectory.<n>BRTS improves over standard OPD on challenging reasoning benchmarks, with the largest gains on harder datasets.
  arXiv  Detail & Related papers  (2026-05-10T19:49:00Z)
• Scaling Reasoning Efficiently via Relaxed On-Policy Distillation [49.39199261602195]
  REOPOLD (Relaxed On-Policy Distillation) is a framework that stabilizes optimization by relaxing the strict imitation constraints of standard on-policy distillation.<n>Specifically, REOPOLD selectively and leverages rewards from the teacher through mixture-based reward clipping, entropy-based token-level dynamic sampling, and a unified exploration-to-refinement training strategy.
  arXiv  Detail & Related papers  (2026-03-11T16:26:52Z)
• Reinforcement-aware Knowledge Distillation for LLM Reasoning [63.53679456364683]
  Reinforcement learning (RL) post-training has recently driven gains in long chain-of-thought reasoning large language models (LLMs)<n>Most existing knowledge distillation methods are designed for supervised fine-tuning (SFT), relying on fixed teacher traces or teacher-student Kullback-Leibler (KL) divergence-based regularization.<n>We propose RL-aware distillation (RLAD), which performs selective imitation during RL -- guiding the student toward the teacher only when it improves the current policy update.
  arXiv  Detail & Related papers  (2026-02-26T00:20:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.