H-OPD: Confidence Aware Heterogeneous Multi-Teacher Multimodal On-policy Distillation

• URL: http://arxiv.org/abs/2607.02592v1
• Date: Wed, 01 Jul 2026 07:37:10 GMT
• Title: H-OPD: Confidence Aware Heterogeneous Multi-Teacher Multimodal On-policy Distillation
• Abstract summary: H-OPD is proposed as a confidence-aware heterogeneous multi-teacher OPD framework for multimodal reasoning.<n>By verifying the complementarity of heterogeneous teachers in the same reasoning process, H-OPD replaces task or sample level teacher routing.<n>H-OPD employs vision-to-language description transfer to enable text-only teachers to access key visual semantics.
• Score: 31.41588745047352
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: On-policy distillation (OPD) has recently emerged as an effective post-training paradigm by providing supervision on student-generated trajectories. However, existing OPD methods for multimodal reasoning usually rely on a static teacher routing, assigning each sample to a single teacher based on modality or task type. This ignores that visual grounding and abstract reasoning may dominate different decoding steps, making a single teacher insufficient for the full trajectory. To this end, H-OPD is proposed as a confidence-aware heterogeneous multi-teacher OPD framework for multimodal reasoning. By verifying the complementarity of heterogeneous teachers in the same reasoning process, H-OPD replaces task or sample level teacher routing with token-level teacher arbitration along the shared student trajectory. H-OPD employs vision-to-language description transfer to enable text-only teachers to access key visual semantics, and uses a confidence-aware arbitration mechanism to dynamically combine vision-language teacher and text-only teachers at each token. Extensive evaluations over 11 widely-used reasoning benchmarks showcase the superior performance of our method.

Related papers

• Every Coin Has Two Sides: On the Dual Nature of Generalization in On-Policy Distillation of Large Language Models [68.52262705081098]
  On-policy distillation transfers teacher capabilities by supervising trajectories sampled from the student's own policy.<n>We find that OPD transfers a teacher's reasoning behavior rather than its answers to particular problems.
  arXiv  Detail & Related papers  (2026-08-17T14:46:53Z)
• REOPD: Reliability-Adaptive Reward Extrapolation for On-Policy Distillation [10.492445451268727]
  REOPD is a reliability-adaptive reward extrapolation framework for on-policy distillation.<n>It preserves teacher alignment while selectively extrapolating along reliable teacher-reference directions.
  arXiv  Detail & Related papers  (2026-08-12T06:15:33Z)
• STEP-OPD: Rethinking Output Targets and Internal Dynamics in On-Policy Distillation for Diffusion Models [14.682283642975271]
  We propose STEP-OPD, an on-policy distillation framework for image generation that extends the student's learning target beyond the teacher.<n>Instead of treating the teacher as the final target, we use the velocity difference between each task-specific teacher and the shared base model as a direction for further learning.<n>In addition, we align the direction and magnitude of representation changes between the student and teacher, enabling the student to learn how representations are progressively transformed.
  arXiv  Detail & Related papers  (2026-08-05T14:11:27Z)
• OPOD: On-Policy Omni Distillation [55.440058537827134]
  Training a single model on pooled multimodal data often fails to match models specialized for individual modalities.<n>We present On-Policy Omni Distillation (OPD), which routes each student response to the matching text, image, or audio teacher.<n>OPD achieves the best average score at every scale, reaching 70.8, 51.7, and 46.2.
  arXiv  Detail & Related papers  (2026-07-23T04:55:04Z)
• dOPSD: On-Policy Self-Distillation for Diffusion Language Models [52.60302464420127]
  Diffusion large language models generate text by iteratively denoising a masked sequence.<n>Supervised fine-tuning is off-policy and suffers from exposure bias, while reinforcement learning gives only sparse, sequence-level rewards.<n>On-policy self-distillation (OPSD) offers a promising alternative, using one model as both student and teacher.
  arXiv  Detail & Related papers  (2026-07-05T17:47:29Z)
• One Student, Many Teachers: Multi-Task On-Policy Distillation via Soft-Prompt Privileged Context [40.91201403670402]
  We propose method, whose teacher differs from the student only by a learnable soft prompt.<n>Method extends naturally to multi-task settings by routing each example in a merged corpus to its corresponding soft-prompt teacher.<n>On Qwen3-1.7B-Base and Phi-4-mini-instruct, the single-task variant matches or exceeds full fine-tuning while training orders of magnitude fewer parameters.
  arXiv  Detail & Related papers  (2026-06-30T19:56:05Z)
• DOPD: Dual On-policy Distillation [59.65986569617147]
  On-policy distillation offers superior capacity transfer by supervising student-sampled trajectories with dense token-level signals.<n>We propose DOPD, an advantage-aware dual distillation paradigm that dynamically routes token-level supervision between privileged teacher and privileged student policies.<n>Experiments on both large language model (LLM) and vision-language model (VLM) settings demonstrate that DOPD consistently outperforms Vanilla OPD.
  arXiv  Detail & Related papers  (2026-06-29T17:55:53Z)
• SAGE-OPD: Selective Agent-Guided Intervention for Multi-Turn On-Policy Distillation [25.25989941933095]
  We propose SAGE-OPD, a verifier-free selective intervention framework specifically designed for multi-turn OPD.<n>Instead of applying teacher supervision uniformly across all turns, SAGE-OPD first observes environment feedback and uses teacher judgment to decide whether each student response should be skipped or intervened on.<n>Experiments on agent tasks show that SAGE-OPD consistently improves over baselines, achieving up to a 13.3% relative improvement in ALFWorld unseen success rate over standard OPD.
  arXiv  Detail & Related papers  (2026-06-17T23:58:14Z)
• OmniOPSD: Rationale-Privileged On-Policy Self-Distillation for Affective Computing [62.62736813348113]
  Reinforcement learning for large language models is often hindered by severe reward sparsity in complex reasoning tasks.<n>We propose OmniOPSD, a framework that uses frontier-generated rationales as teacher-side privileged evidence.<n>Experiments on MER-UniBench show that OmniOPSD achieves state-of-the-art performance with an average score of $84.19$.
  arXiv  Detail & Related papers  (2026-06-14T16:51:22Z)
• Breaking the Tokenizer Barrier: On-Policy Distillation across Model Families [32.34854415784466]
  On-Policy Distillation (OPD) has become a core technique in the post-training of Large Language Models (LLMs) for transferring knowledge from domain experts to student models.<n>Existing OPD distillation methods require teacher and student models to share the same tokenizer, restricting the applicability of OPD within the model series.<n>In this work, we enable the standard on-policy distillation method to operate across model families, ensuring that high-fidelity token-level signals can propagate across different tokenizers.
  arXiv  Detail & Related papers  (2026-06-08T13:12:01Z)
• Prefix Teach, Suffix Fade: Local Teachability Collapse in Strong-to-Weak On-Policy Distillation [49.117085054884676]
  On-policy distillation trains a student model on its own rollouts using dense feedback from a stronger teacher.<n>We operationalize this principle through a trajectory-specific release rule.<n> Experimental results across strong-to-weak distillation tasks indicate that this release rule consistently outperforms standard full-trajectory OPD.
  arXiv  Detail & Related papers  (2026-05-13T15:05:30Z)
• Teacher-Guided Policy Optimization for LLM Distillation [90.49982387646861]
  Teacher-Guided Policy Optimization (TGPO) is an on-policy algorithm that incorporates dense directional guidance by leveraging teacher predictions conditioned on the student's rollout.<n>Experiments on complex reasoning benchmarks demonstrate that TGPO significantly outperforms standard baselines and is robust to different teachers.
  arXiv  Detail & Related papers  (2026-05-13T09:20:03Z)
• Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe [53.40076304466524]
  We propose Uni-OPD, a unified OPD framework that generalizes across Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs)<n>Specifically, from the student's perspective, we adopt two data balancing strategies to promote exploration of informative student-generated states during training.<n>We develop an outcome-guided margin calibration mechanism to restore order consistency between correct and incorrect trajectories.
  arXiv  Detail & Related papers  (2026-05-05T12:15:21Z)
• "The Whole Is Greater Than the Sum of Its Parts": A Compatibility-Aware Multi-Teacher CoT Distillation Framework [16.96094045628127]
  Chain-of-Thought (CoT) reasoning empowers Large Language Models (LLMs) with remarkable capabilities but typically requires prohibitive parameter scales.<n>CoT distillation has emerged as a promising paradigm to transfer reasoning prowess into compact Student Models (SLMs)<n>We introduce COMPACT, a framework that adaptively fuses supervisions from different teachers by dynamically weighting teacher gradients.
  arXiv  Detail & Related papers  (2026-01-20T14:05:19Z)
• Heuristic-Free Multi-Teacher Learning [0.6597195879147557]
  Teacher2Task is a novel framework for multi-teacher learning that eliminates the need for manual aggregations.<n>Instead of relying on aggregated labels, the framework transforms the training data, consisting of ground truth labels and annotations from N teachers, into N+1 distinct tasks.
  arXiv  Detail & Related papers  (2024-11-19T18:45:16Z)
• Representation Consolidation for Training Expert Students [54.90754502493968]
  We show that a multi-head, multi-task distillation method is sufficient to consolidate representations from task-specific teacher(s) and improve downstream performance. Our method can also combine the representational knowledge of multiple teachers trained on one or multiple domains into a single model.
  arXiv  Detail & Related papers  (2021-07-16T17:58:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.