OPOD: On-Policy Omni Distillation

• URL: http://arxiv.org/abs/2607.20918v2
• Date: Fri, 24 Jul 2026 07:07:42 GMT
• Title: OPOD: On-Policy Omni Distillation
• Abstract summary: Training a single model on pooled multimodal data often fails to match models specialized for individual modalities.<n>We present On-Policy Omni Distillation (OPD), which routes each student response to the matching text, image, or audio teacher.<n>OPD achieves the best average score at every scale, reaching 70.8, 51.7, and 46.2.
• Score: 55.440058537827134
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Omni-modal models can handle text, images, and audio in one system, but improving all of these abilities together remains difficult. Training a single model on pooled multimodal data often fails to match models specialized for individual modalities. On-policy distillation (OPD) offers a way to combine such specialists: the student generates a response, and a teacher evaluates that same response, so the student learns directly from behaviors it actually produces. Yet using several teachers can introduce competing guidance and improve one modality at the expense of another. We present On-Policy Omni Distillation (OPOD), which routes each student response to the matching text, image, or audio teacher. OPOD keeps teacher guidance only on tokens where the teacher assigns a higher probability than the student, adjusts the influence of each modality teacher independently during training, and asks the routed teacher to assess both the final answer and whether the reasoning supports the correct answer. Across twelve benchmarks and three backbone sizes, OPOD achieves the best average score at every scale, reaching 70.8, 51.7, and 46.2 and exceeding the strongest comparator by 2.1, 1.8, and 1.7 points. On the 30B model, it outperforms both the base model and a counterpart post-trained jointly on pooled multimodal data on all twelve benchmarks, and ranks first or second on eleven even when the individual specialists are included. The specialists are discarded after training, leaving one deployable omni-modal model. These results show that coordinating modality-specific teachers is an effective way to improve a shared model while maintaining cross-modal balance.

Related papers

• Self-OPD: On-Policy Distillation for Flow Matching Models without Teacher [36.07614733558147]
  We introduce textbfSelf-OPD, a teacher-free OPD framework for flow matching models.<n>Self-OPD branches the deterministic next-state prediction into $K$ SDE candidates, rolls them out with the ODE sampler, and compares their rewards against a deterministic self-reference baseline.<n>Experiments on single and mixed reward benchmarks show that Self-OPD outperforms prior RL and OPD methods without task-specific teachers.
  arXiv  Detail & Related papers  (2026-08-27T09:34:23Z)
• Any-OPD: Heterogeneous On-Policy Distillation for Flow-Matching Models via Representation-Space Bridging [13.653084100002737]
  On-policy distillation in which a teacher corrects samples that the student itself generates presupposes that the two models speak the same language.<n>We present Any-OPD, to our knowledge the first framework for on-policy distillation between arbitrary pairs of latent flow-matching generators.
  arXiv  Detail & Related papers  (2026-08-04T08:23:57Z)
• Beyond the Best Teacher: Expanding and Compressing the Reasoning Solution Manifold [4.090731563364332]
  We argue that reinforcement learning policies should be viewed as local probes of a multi-basin reasoning solution manifold.<n>We propose an expand-then-compress framework that couples teacher construction with multi-teacher policy distillation.
  arXiv  Detail & Related papers  (2026-07-30T07:05:37Z)
• H-OPD: Confidence Aware Heterogeneous Multi-Teacher Multimodal On-policy Distillation [31.41588745047352]
  H-OPD is proposed as a confidence-aware heterogeneous multi-teacher OPD framework for multimodal reasoning.<n>By verifying the complementarity of heterogeneous teachers in the same reasoning process, H-OPD replaces task or sample level teacher routing.<n>H-OPD employs vision-to-language description transfer to enable text-only teachers to access key visual semantics.
  arXiv  Detail & Related papers  (2026-07-01T07:37:10Z)
• On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agents [50.78688247330029]
  On-Policy Distillation (OPD) is a natural recipe for transferring such capabilities to smaller students.<n>We propose Guided On-Policy Distillation (Guided-OPD), a simple yet effective algorithm that mixes teacher- and student-generated turns within each rollout.<n>Guided-OPD improves Score by 21.1% and Success Rate by 25.5% over vanilla OPD on average.
  arXiv  Detail & Related papers  (2026-06-14T16:41:45Z)
• Maestro: Reinforcement Learning to Orchestrate Hierarchical Model-Skill Ensembles [32.54156025863882]
  We present Maestro, a Reinforcement Learning-driven orchestration framework.<n>Maestro reframes heterogeneous multimodal tasks as a sequential decision-making process over a hierarchical model-skill registry.<n>We evaluate Maestro across ten representative multimodal benchmarks spanning mathematical reasoning, chart understanding, high-resolution perception, and domain-specific analysis.
  arXiv  Detail & Related papers  (2026-05-21T08:47:49Z)
• When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoning [45.79647925282674]
  On-policy self-distillation (OPSD) trains a student on its own rollouts using a privileged teacher.<n>Existing entropy-based OPD methods relax this uniformity by modulating token-level supervision with teacher entropy.<n>We propose Position-Weighted On-Policy Self-Distillation (PW-OPSD), which applies an increasing position weight while keeping the same student rollout, privileged teacher pass, and clipped forward-KL target as OPSD.
  arXiv  Detail & Related papers  (2026-05-20T18:14:03Z)
• Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe [53.40076304466524]
  We propose Uni-OPD, a unified OPD framework that generalizes across Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs)<n>Specifically, from the student's perspective, we adopt two data balancing strategies to promote exploration of informative student-generated states during training.<n>We develop an outcome-guided margin calibration mechanism to restore order consistency between correct and incorrect trajectories.
  arXiv  Detail & Related papers  (2026-05-05T12:15:21Z)
• UCO: A Multi-Turn Interactive Reinforcement Learning Method for Adaptive Teaching with Large Language Models [59.693733170193944]
  Large language models (LLMs) are shifting from answer providers to intelligent tutors in educational settings.<n>Recent reinforcement learning approaches address this limitation but face two critical challenges.<n>We propose the Unidirectional Cognitive Optimization (UCO) method to address these challenges.
  arXiv  Detail & Related papers  (2025-11-12T01:27:02Z)
• Find Your Optimal Teacher: Personalized Data Synthesis via Router-Guided Multi-Teacher Distillation [47.814833568523255]
  PerSyn operates under a new Route then Generate'' paradigm to create data tailored to each student model.<n>Experiments across different model families and scales demonstrate that PerSyn consistently achieves superior or comparable performance.
  arXiv  Detail & Related papers  (2025-10-13T02:36:36Z)
• More Than One Teacher: Adaptive Multi-Guidance Policy Optimization for Diverse Exploration [103.1589018460702]
  "guidance-on-demand" approach expands exploration while preserving the value of self-discovery.<n>Experiments show AMPO substantially outperforms a strong baseline.<n>Using four peer-sized teachers, our method achieves comparable results to approaches that leverage a single, more powerful teacher.
  arXiv  Detail & Related papers  (2025-10-02T17:14:00Z)
• Merge-of-Thought Distillation [23.53356244978525]
  Merge-of-Thought Distillation (MoT) is a lightweight framework that alternates between teacher-specific supervised fine-tuning branches and weight-space merging the resulting student variants.<n>On competition math benchmarks, applying MoT to a Qwen3-14B student surpasses strong models including Deepseek-R1, Qwen3-32B, and OpenAI-O1.<n>MoT consistently outperforms the best single-teacher distillation, improves general reasoning beyond mathematics, and shows robustness to distribution-shifted and peer-level teachers.
  arXiv  Detail & Related papers  (2025-09-10T17:46:57Z)
• Learning to Teach with Student Feedback [67.41261090761834]
  Interactive Knowledge Distillation (IKD) allows the teacher to learn to teach from the feedback of the student. IKD trains the teacher model to generate specific soft target at each training step for a certain student. Joint optimization for both teacher and student is achieved by two iterative steps.
  arXiv  Detail & Related papers  (2021-09-10T03:01:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.