Filter, Then Reweight: Rethinking Optimization Granularity in On-Policy Distillation

• URL: http://arxiv.org/abs/2606.02684v2
• Date: Thu, 04 Jun 2026 15:52:50 GMT
• Title: Filter, Then Reweight: Rethinking Optimization Granularity in On-Policy Distillation
• Abstract summary: On-Policy distillation (OPD) in large language models is shifting from full-trace KL supervision toward more selective training paradigms.<n>Recent OPD methods increasingly focus on selecting which trajectories to learn from, which tokens are most informative, and which supervision signals are most reliable.<n>We propose fireicon FiRe-OPD (Filter, then Reweight), which jointly adjusts supervision signals at both trajectory and token levels.
• Score: 16.086812358369798
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: On-Policy distillation (OPD) in large language models is shifting from full-trace KL supervision toward more selective training paradigms. Recent OPD methods increasingly focus on selecting which trajectories to learn from, which tokens are most informative, and which supervision signals are most reliable. Motivated by this trend, we rethink optimization granularity of OPD and propose \fireicon\ FiRe-OPD (Filter, then Reweight), which jointly adjusts supervision signals at both trajectory and token levels. In details, FiRe-OPD first filters trajectories to remove low-quality rollout samples, and then applies soft reweighting within the retained trajectories to emphasize informative tokens. Compared with hard token selection, FiRe-OPD leverages a soft-weighting mechanism to effectively mitigate information loss and enhance optimization stability, thereby achieving finer-grained OPD optimization. We validate the effectiveness of FiRe-OPD across strong-to-weak, single-teacher, and multi-teacher settings, and demonstrate its superiority over recent token-level OPD methods ( (e.g., +6.25 on AIME 2024 in strong-to-weak, +18.81 on Miner in multi-teacher). Our code is available at https://github.com/YuYingLi0/FiRe-OPD.

Related papers

• When Teacher Guidance Misleads: Reward-Aligned On-Policy Distillation [21.86522736658343]
  We propose Reward-Aligned On-Policy Distillation (RA-OPD) to mitigate misaligned teacher guidance.<n> RA-OPD selects more reliable trajectories to improve student model performance without requiring additional computational cost.<n>We evaluate RA-OPD on math and code benchmarks using models from the Qwen3 family and the DeepSeek-R1 family.
  arXiv  Detail & Related papers  (2026-08-28T06:05:50Z)
• When Teachers Mislead: Spurious-Signal-Aware On-Policy Distillation [47.38632303827905]
  On-Policy Distillation (OPD) transfers teacher capabilities by supervising student-sampled trajectories with dense token-level teacher signals.<n>We propose SA-OPD, a Spurious-Signal-Aware On-Policy Distillation framework that identifies and filters misleading token-level supervision.
  arXiv  Detail & Related papers  (2026-08-04T13:17:30Z)
• Distill Where You Fail: Recovering Learning Signals of Negative RL-Groups from Adaptive Teacher Guidance [50.77757355236912]
  We propose RSTG (Recovering Learning Signals via Adaptive Teacher Guidance), which applies distillation selectively and precisely where it matters most.<n> Experiments demonstrate that RSTG substantially outperforms naive GRPO+OPD by +4.02% on math and+3.05% on code.
  arXiv  Detail & Related papers  (2026-08-01T17:29:14Z)
• On the Position Bias of On-Policy Distillation [19.36735875165262]
  We show that Importance-Weighted On-Policy Distillation (IW-OPD) converges significantly faster than On-Policy Distillation (OPD)<n>IW-OPD achieves better final performance than standard OPD in both same-size and cross-scale settings, improving performance up to 6.9 points on AIME-2025.
  arXiv  Detail & Related papers  (2026-06-21T17:20:21Z)
• Stabilizing On-Policy Distillation for MLLM Reasoning with Global Normalization [9.860371691501076]
  On-policy distillation (OPD) has recently emerged as an important post-training paradigm.<n>We propose Globally Normalized Distillation Policy Optimization (GNDPO), a practical method that stabilizes optimization by transforming raw KL scores into batch-level relative advantages.
  arXiv  Detail & Related papers  (2026-06-08T06:41:31Z)
• Learning to Foresee: Unveiling the Unlocking Efficiency of On-Policy Distillation [51.210887267509854]
  We argue that OPD's efficiency stems from a form of foresight'': it establishes a stable update trajectory toward the final model early in training.<n>We propose textbfEffOPD, a plug-and-play acceleration method that speeds up OPD by adaptively selecting an extrapolation step size and moving along the current update direction.
  arXiv  Detail & Related papers  (2026-05-12T08:19:15Z)
• MASS-DPO: Multi-negative Active Sample Selection for Direct Policy Optimization [66.81594613265833]
  Multi-negative preference optimization under the Plackett--Luce (PL) model extends Direct Preference Optimization (DPO)<n>We introduce MASS-DPO, a multi-negative active sample selection method that derives a PL-specific Fisher-information objective for selecting compact, informative negative subsets within each prompt.<n> MASS-DPO consistently exceeds or matches existing methods in accuracy, improves Recall/NDCG and margin-based optimization dynamics, and delivers stronger alignment with substantially fewer negatives.
  arXiv  Detail & Related papers  (2026-05-11T16:18:08Z)
• GradAlign: Gradient-Aligned Data Selection for LLM Reinforcement Learning [55.03441672267886]
  We propose GradAlign, a gradient-aligned data selection method for reinforcement learning.<n>We evaluate GradAlign across three data regimes: unreliable reward signals, distribution imbalance, and low-utility training corpus.
  arXiv  Detail & Related papers  (2026-02-25T01:54:50Z)
• Supervised Fine Tuning on Curated Data is Reinforcement Learning (and can be improved) [3.13388270461847]
  We draw on a connection between supervised fine-tuning (SFT) and the theory and practice of finding optimal policies via Reinforcement Learning (RL)<n>We show that a small modification to SFT leads to an importance weighted variant that behaves closer to training with RL as it.<n>We refer to this variant as importance weighted supervised fine-tuning (iw-SFT)
  arXiv  Detail & Related papers  (2025-07-17T07:26:54Z)
• Active Learning for Direct Preference Optimization [59.84525302418018]
  Direct preference optimization (DPO) is a form of reinforcement learning from human feedback.<n>We propose an active learning framework for DPO, which can be applied to collect human feedback online or to choose the most informative subset of already collected feedback offline.
  arXiv  Detail & Related papers  (2025-03-03T00:36:31Z)
• RaCT: Ranking-aware Chain-of-Thought Optimization for LLMs [30.216174551427443]
  Large language models (LLMs) have demonstrated remarkable potential in text reranking tasks.<n> conventional supervised fine-tuning approaches for specializing LLMs in ranking tasks often lead to significant degradation of the models' general-purpose abilities.<n>This paper presents a novel methodology that strategically combines Chain-of-Thought (CoT) prompting techniques with an innovative two-stage training pipeline.
  arXiv  Detail & Related papers  (2024-12-18T23:24:15Z)
• Permutative Preference Alignment from Listwise Ranking of Human Judgments [40.23480751285947]
  We develop an end-to-end alignment algorithm by approximating NDCG with a differentiable surrogate loss.<n>We show that NDCG-based approaches improve ranking accuracy more effectively than B-T-based methods.
  arXiv  Detail & Related papers  (2024-10-06T03:49:28Z)
• Self-supervised Preference Optimization: Enhance Your Language Model with Preference Degree Awareness [27.43137305486112]
  We propose a novel Self-supervised Preference Optimization (SPO) framework, which constructs a self-supervised preference degree loss combined with the alignment loss. The results demonstrate that SPO can be seamlessly integrated with existing preference optimization methods to achieve state-of-the-art performance.
  arXiv  Detail & Related papers  (2024-09-26T12:37:26Z)
• OPTune: Efficient Online Preference Tuning [107.44836901099]
  We propose a more efficient data exploration strategy for online preference tuning (OPTune) OPTune dynamically samples informative responses for on-policy preference alignment. In our evaluations, OPTune'd LLMs enjoy 1.27-1.56x faster training speed due to the efficient data exploration strategy.
  arXiv  Detail & Related papers  (2024-06-11T18:55:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.