OPD-Aha: From Linguistic Momentum to Visual Reflection in Multimodal On-Policy Distillation

• URL: http://arxiv.org/abs/2609.16459v1
• Date: Tue, 15 Sep 2026 00:25:28 GMT
• Title: OPD-Aha: From Linguistic Momentum to Visual Reflection in Multimodal On-Policy Distillation
• Abstract summary: Privileged on-policy distillation improves multimodal reasoning by allowing a teacher to evaluate student trajectories using rich, training-only visual evidence.<n>We find that the teacher's visual corrective preference is not lost under this misleading agreement.<n>We introduce OPD-Aha, which reconstructs the distillation target directly from this isolated visual preference.
• Score: 15.986808086256792
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Privileged on-policy distillation improves multimodal reasoning by allowing a teacher to evaluate student trajectories using rich, training-only visual evidence. Both models score these trajectories while conditioning on the same student-generated prefix. When a student misinterprets an image early in a response, this accumulating erroneous rationale eventually pulls the teacher away from its visual evidence. The teacher and student converge on the same hallucination, causing standard cross-model supervision to collapse precisely where correction is most needed. We find that the teacher's visual corrective preference is not lost under this misleading agreement. Comparing the predictions of the identical teacher given the real image and a visual null reveals that the privileged evidence still pushes the model toward the correct interpretation. We introduce OPD-Aha, which reconstructs the distillation target directly from this isolated visual preference rather than relying on the fragile teacher-student discrepancy. This reconstructed target aggressively suppresses continuations that contradict the image. Trained with this objective, students learn to naturally interrupt their own flawed reasoning with reflection tokens such as wait and actually. After reflection, subsequent generation relies less on the accumulated erroneous text and more on the visual evidence. Correcting these trajectories mid-generation fundamentally alters the reasoning process, yielding broad and consistent improvements across diverse fine-grained perception and complex multimodal reasoning benchmarks. Our code and models are available at https://github.com/Echochef/OPD-Aha.

Related papers

• One Symptom, Three Levers: A Critical Review of On-Policy Self-Distillation [3.0867286767516937]
  On-policy distillation trains a language model on its own generations while a teacher scores them token by token.<n>The teacher is the model itself, conditioned on privileged information the student will not have at test time.<n>Early results were promising, with accuracy comparable to reinforcement learning at a fraction of the generated tokens.<n>But the same asymmetry that produces the signal also biases it.
  arXiv  Detail & Related papers  (2026-08-26T15:52:19Z)
• VAD: Attributing Visual Evidence for Target Reconstruction in Multimodal On-Policy Distillation [46.347555076026964]
  Multimodal on-policy distillation transfers fine-grained visual knowledge by supervising student-generated trajectories with a privileged-view teacher.<n>Key challenge is to estimate which corrections are supported by visual evidence, not merely where or how strongly to distill.<n>We introduce Visual Attribution Distillation (VAD), a counterfactual target-reconstruction algorithm that estimates the visually attributable part of a teacher correction.
  arXiv  Detail & Related papers  (2026-07-30T17:43:46Z)
• H$^2$SD: Hybrid Hindsight Self-Distillation [61.71131241473028]
  Reinforcement learning with verifiable rewards (RLVR) provides reliable outcome supervision for language model reasoning.<n>Existing self-distillation methods add a privileged teacher but typically assign it a fixed role.<n>We introduce Hybrid Hindsight Self-Distillation ($mathrmH2mathrmSD$), which jointly adapts teacher context and update strategy to trajectory correctness.
  arXiv  Detail & Related papers  (2026-07-21T10:47:27Z)
• Geometric Self-Distillation for Reasoning Generalization [12.38444431260744]
  On-policy distillation is a practical post-training recipe for large language models.<n>In privileged-context self-distillation, teacher and student are the same model conditioned on the same prefix.<n>We introduce GeoSD, a geometric self-distillation objective that treats this drift as movement in the student's predictive behavior.
  arXiv  Detail & Related papers  (2026-07-07T23:16:19Z)
• Adaptive Teacher Exposure for Self-Distillation in LLM Reasoning [2.9029480541171107]
  We propose adaptive teacher exposure as an effective new axis for reasoning self-distillation.<n>We show that Adaptive Teacher Exposure for Self-Distillation (ATESD) consistently outperforms competitive self-distillation and RL baselines.
  arXiv  Detail & Related papers  (2026-05-12T03:15:58Z)
• On-Policy Distillation with Best-of-N Teacher Rollout Selection [54.91780727674628]
  We propose BRTS, a Best-of-N Rollout Teacher Selection framework for on-policy distillation.<n>BRTS augments standard student-context OPD with a teacher-context supervision branch constructed from the teacher trajectory.<n>BRTS improves over standard OPD on challenging reasoning benchmarks, with the largest gains on harder datasets.
  arXiv  Detail & Related papers  (2026-05-10T19:49:00Z)
• Long-Chain Reasoning Distillation via Adaptive Prefix Alignment [57.130176131042965]
  We propose a framework that exploits teacher CoTs for distillation through adaptive prefix alignment.<n>P-ALIGN adaptively truncates teacher-generated reasoning trajectories by determining whether the remaining suffix is concise.<n>Experiments on multiple mathematical reasoning benchmarks demonstrate that P-ALIGN outperforms all baselines by over 3%.
  arXiv  Detail & Related papers  (2026-01-15T04:40:45Z)
• DualFete: Revisiting Teacher-Student Interactions from a Feedback Perspective for Semi-supervised Medical Image Segmentation [12.629975999398303]
  This work introduces a feedback mechanism into the teacher-student framework to counteract error reconfirmations.<n>We specify this interaction hinges on two key components: the feedback attributor, which designates pseudo-labels triggering the student's update, and the feedback receiver, which determines where to apply this feedback.<n>Building on this, a dual-teacher feedback model is further proposed, which allows more dynamics in the feedback loop and fosters more gains.
  arXiv  Detail & Related papers  (2025-11-12T13:29:24Z)
• Learning to Focus: Causal Attention Distillation via Gradient-Guided Token Pruning [62.23671919314693]
  Large language models (LLMs) have demonstrated significant improvements in contextual understanding.<n>However, their ability to attend to truly critical information during long-context reasoning and generation still falls behind the pace.<n>We introduce a two-stage framework called Learning to Focus (LeaF) to mitigate confounding factors.
  arXiv  Detail & Related papers  (2025-06-09T15:16:39Z)
• SuperCorrect: Advancing Small LLM Reasoning with Thought Template Distillation and Self-Correction [89.56181323849512]
  SuperCorrect is a novel two-stage framework that uses a large teacher model to supervise and correct both the reasoning and reflection processes of a smaller student model.<n>In the first stage, we extract hierarchical high-level and detailed thought templates from the teacher model to guide the student model in eliciting more fine-grained reasoning thoughts.<n>In the second stage, we introduce cross-model collaborative direct preference optimization (DPO) to enhance the self-correction abilities of the student model.
  arXiv  Detail & Related papers  (2024-10-11T17:25:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.