When Teachers Mislead: Spurious-Signal-Aware On-Policy Distillation

• URL: http://arxiv.org/abs/2608.03632v1
• Date: Tue, 04 Aug 2026 13:17:30 GMT
• Title: When Teachers Mislead: Spurious-Signal-Aware On-Policy Distillation
• Abstract summary: On-Policy Distillation (OPD) transfers teacher capabilities by supervising student-sampled trajectories with dense token-level teacher signals.<n>We propose SA-OPD, a Spurious-Signal-Aware On-Policy Distillation framework that identifies and filters misleading token-level supervision.
• Score: 47.38632303827905
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: On-Policy distillation (OPD) transfers teacher capabilities by supervising student-sampled trajectories with dense token-level teacher signals. Recent selective OPD methods improve this process by prioritizing signals that are confident, informative, or learnable. However, the assumptions overlook a fundamental failure mode of language models: their token-level judgments can be driven by input-agnostic language priors, formatting conventions, or stereotyped reasoning templates rather than task-specific evidence. We refer to such optimization-relevant but weakly input-grounded supervision as spurious signals in OPD, which may produce large gradients while contributing little task-improving direction. To mitigate this issue, we propose SA-OPD, a Spurious-Signal-Aware On-Policy Distillation framework that identifies and filters misleading token-level supervision based on input-groundedness and optimization impact. SA-OPD introduces a lightweight input-groundedness proxy estimating whether a token-level distillation signal truly depends on the input. It then filters only tokens that simultaneously exhibit low input-groundedness and extreme distillation divergence, thereby removing high-impact spurious updates and achieving fine-grained OPD optimization. Extensive experiments on both large language model (LLM) and vision-language model (VLM) settings demonstrate that SA-OPD consistently outperforms Vanilla OPD and competitive selective methods. These results establish input-groundedness as a key dimension for OPD supervision selection and offer a simple, effective strategy for mitigating spurious updates.

Related papers

• OPD-V: Visual On-Policy Self-Distillation with Modality Balance [60.972871141106886]
  On-Policy Self-Distillation (OPSD) has become a standard post-training approach for improving visual reasoning in large language models (MLLMs)<n>Existing methods draw privileged information from diverse input sources to guide self-distillation.<n>We introduce OPD-V, a visual OPSD paradigm that instantiates such information through the Positive Teacher and Negative Teacher.
  arXiv  Detail & Related papers  (2026-08-05T17:53:06Z)
• dOPSD: On-Policy Self-Distillation for Diffusion Language Models [52.60302464420127]
  Diffusion large language models generate text by iteratively denoising a masked sequence.<n>Supervised fine-tuning is off-policy and suffers from exposure bias, while reinforcement learning gives only sparse, sequence-level rewards.<n>On-policy self-distillation (OPSD) offers a promising alternative, using one model as both student and teacher.
  arXiv  Detail & Related papers  (2026-07-05T17:47:29Z)
• Filter, Then Reweight: Rethinking Optimization Granularity in On-Policy Distillation [16.086812358369798]
  On-Policy distillation (OPD) in large language models is shifting from full-trace KL supervision toward more selective training paradigms.<n>Recent OPD methods increasingly focus on selecting which trajectories to learn from, which tokens are most informative, and which supervision signals are most reliable.<n>We propose fireicon FiRe-OPD (Filter, then Reweight), which jointly adjusts supervision signals at both trajectory and token levels.
  arXiv  Detail & Related papers  (2026-06-01T17:58:22Z)
• Uncertainty-Aware Exploratory Direct Preference Optimization for Multimodal Large Language Models [53.15468578562038]
  We propose an Uncertainty-aware Exploratory Direct Preference Optimization (UE-DPO) method for Multimodal Large Language Models (MLLMs)<n>We first quantify the uncertainty from the model's failure to ground token predictions in the given image.<n>We then encourage more learning pressure on visually deficient tokens in preferred samples, and alleviate the over-penalization of beneficial knowledge in dispreferred samples.
  arXiv  Detail & Related papers  (2026-05-06T13:08:12Z)
• Direct Diffusion Score Preference Optimization via Stepwise Contrastive Policy-Pair Supervision [14.612317970237436]
  Diffusion models have achieved impressive results in generative tasks such as text-to-image synthesis.<n>They often struggle to fully align outputs with nuanced user intent and maintain consistent aesthetic quality.<n>Existing preference-based training methods help address these issues but rely on costly and potentially noisy human datasets.
  arXiv  Detail & Related papers  (2025-12-29T12:46:07Z)
• Token-Importance Guided Direct Preference Optimization [2.230951739798399]
  We propose a Token-Importance Guided Direct Preference Optimization (TI-DPO) to ensure that large language models generate outputs aligned with human preferences.<n> Experimental results show that TI-DPO achieves higher accuracy and stronger generative diversity, providing more stable and computationally efficient solutions.
  arXiv  Detail & Related papers  (2025-05-26T08:11:24Z)
• The First Few Tokens Are All You Need: An Efficient and Effective Unsupervised Prefix Fine-Tuning Method for Reasoning Models [69.798277882245]
  We introduce Unsupervised Prefix Fine-Tuning (UPFT) to enhance large language models' reasoning efficiency.<n>UPFT removes the need for labeled data or exhaustive sampling.<n> Experiments show that UPFT matches the performance of supervised methods.
  arXiv  Detail & Related papers  (2025-03-04T18:56:03Z)
• Debiasing Multimodal Large Language Models via Penalization of Language Priors [38.97645845493758]
  Multimodal Large Language Models (MLLMs) have become indispensable tools in computer vision and natural language processing.<n>Despite their advancements, our investigation reveals a noteworthy bias: the generated content is often driven more by the inherent priors of the underlying Large Language Models (LLMs) than by the input image.<n>We propose two simple, training-free strategies to rectify these biases and redirect the model's focus toward visual information.
  arXiv  Detail & Related papers  (2024-03-08T12:35:07Z)
• Approximated Prompt Tuning for Vision-Language Pre-trained Models [54.326232586461614]
  In vision-language pre-trained models, prompt tuning often requires a large number of learnable tokens to bridge the gap between the pre-training and downstream tasks. We propose a novel Approximated Prompt Tuning (APT) approach towards efficient VL transfer learning.
  arXiv  Detail & Related papers  (2023-06-27T05:43:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.