One Student, Many Teachers: Multi-Task On-Policy Distillation via Soft-Prompt Privileged Context

• URL: http://arxiv.org/abs/2607.18293v1
• Date: Tue, 30 Jun 2026 19:56:05 GMT
• Title: One Student, Many Teachers: Multi-Task On-Policy Distillation via Soft-Prompt Privileged Context
• Abstract summary: We propose method, whose teacher differs from the student only by a learnable soft prompt.<n>Method extends naturally to multi-task settings by routing each example in a merged corpus to its corresponding soft-prompt teacher.<n>On Qwen3-1.7B-Base and Phi-4-mini-instruct, the single-task variant matches or exceeds full fine-tuning while training orders of magnitude fewer parameters.
• Score: 40.91201403670402
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: On-policy self-distillation (OPSD) teaches large language models new skills through a teacher that shares the student's backbone and supervises its own rollouts. Existing teachers either inject privileged context at the input -- inducing post-hoc rationalization -- or fine-tune weights, accumulating drift and forgetting across tasks. We propose \method, whose teacher differs from the student only by a learnable soft prompt: trained on $(x, y_\text{gold})$ pairs with the backbone frozen, the prompt yields a task-specific teacher that preserves the student's exact representational geometry. \method\ extends naturally to multi-task settings by routing each example in a merged corpus to its corresponding soft-prompt teacher, allowing a single student to absorb knowledge from $K$ teachers in parallel; at inference, all prompts are discarded. On Qwen3-1.7B-Base and Phi-4-mini-instruct across four tasks (Science, Tool Use, Biology, Math), the single-task variant (OPD with a PT teacher) matches or exceeds full fine-tuning while training orders of magnitude fewer parameters, and the multi-task variant achieves the best overall average ($56.2$ on Qwen3-1.7B-Base) while preserving general-capability benchmarks -- in contrast to sequential SFT, which degrades both.

Related papers

• ReOrder-OPD:Reliability-Aware Prompt Ordering for On-Policy Distillation [25.69241346799495]
  We define prompt-level teacher continuation reliability $R$ as the teacher's probability of reaching a correct answer from a student prefix.<n>We show that high-$R$ prompts yield larger OPD gains and that descending-$R$ training outperforms random and ascending orders.
  arXiv  Detail & Related papers  (2026-08-11T13:27:56Z)
• Beyond the Best Teacher: Expanding and Compressing the Reasoning Solution Manifold [4.090731563364332]
  We argue that reinforcement learning policies should be viewed as local probes of a multi-basin reasoning solution manifold.<n>We propose an expand-then-compress framework that couples teacher construction with multi-teacher policy distillation.
  arXiv  Detail & Related papers  (2026-07-30T07:05:37Z)
• OPOD: On-Policy Omni Distillation [57.03701765491103]
  Training a single model on pooled multimodal data often fails to match models specialized for individual modalities.<n>We present On-Policy Omni Distillation (OPD), which routes each student response to the matching text, image, or audio teacher.<n>OPD achieves the best average score at every scale, reaching 70.8, 51.7, and 46.2.
  arXiv  Detail & Related papers  (2026-07-23T04:55:04Z)
• Multi-Turn On-Policy Distillation with Prefix Replay [73.10000453999088]
  We propose Replayed-Prefix On-Policy Distillation (ReOPD) for agentic tasks.<n>ReOPD reuses pre-collected teacher trajectories as replayed prefixes.<n>It preserves or improves OPD-level accuracy, uses zero tool calls during student training, and is at least 4$times$ faster per training step than OPD.
  arXiv  Detail & Related papers  (2026-07-06T07:56:53Z)
• H-OPD: Confidence Aware Heterogeneous Multi-Teacher Multimodal On-policy Distillation [31.41588745047352]
  H-OPD is proposed as a confidence-aware heterogeneous multi-teacher OPD framework for multimodal reasoning.<n>By verifying the complementarity of heterogeneous teachers in the same reasoning process, H-OPD replaces task or sample level teacher routing.<n>H-OPD employs vision-to-language description transfer to enable text-only teachers to access key visual semantics.
  arXiv  Detail & Related papers  (2026-07-01T07:37:10Z)
• Teaching Prompts to Coordinate: Hierarchical Layer-Grouped Prompt Tuning for Continual Learning [69.17264556340244]
  We propose a novel hierarchical layer-grouped prompt tuning method for continual learning.<n>It improves model stability in two ways: (i) Layers in the same group share roughly the same prompts, which are adjusted by position encoding.<n>It utilizes a single task-specific root prompt to learn to generate sub-prompts for each layer group.
  arXiv  Detail & Related papers  (2025-11-15T08:15:51Z)
• Heuristic-Free Multi-Teacher Learning [0.6597195879147557]
  Teacher2Task is a novel framework for multi-teacher learning that eliminates the need for manual aggregations.<n>Instead of relying on aggregated labels, the framework transforms the training data, consisting of ground truth labels and annotations from N teachers, into N+1 distinct tasks.
  arXiv  Detail & Related papers  (2024-11-19T18:45:16Z)
• PECTP: Parameter-Efficient Cross-Task Prompts for Incremental Vision Transformer [76.39111896665585]
  Incremental Learning (IL) aims to learn deep models on sequential tasks continually. Recent vast pre-trained models (PTMs) have achieved outstanding performance by prompt technique in practical IL without the old samples.
  arXiv  Detail & Related papers  (2024-07-04T10:37:58Z)
• Parameterizing Context: Unleashing the Power of Parameter-Efficient Fine-Tuning and In-Context Tuning for Continual Table Semantic Parsing [13.51721352349583]
  This paper introduces a novel method integrating textitcontext-efficient fine-tuning (PEFT) and textitin-adaptive tuning (ICT) for training a continual table semantic parsing. The teacher addresses the few-shot problem using ICT, which procures contextual information by demonstrating a few training examples. In turn, the student leverages the proposed PEFT framework to learn from the teacher's output distribution, and subsequently compresses and saves the contextual information to the prompts, eliminating the need to store any training examples.
  arXiv  Detail & Related papers  (2023-10-07T13:40:41Z)
• TGRL: An Algorithm for Teacher Guided Reinforcement Learning [45.38447023752256]
  It is common to train a policy to maximize a combination of reinforcement and teacher-student learning objectives. We present a $textitprincipled$ approach, along with an approximate implementation for $textitdynamically$ and $textitautomatically$ balancing when to follow the teacher and when to use rewards.
  arXiv  Detail & Related papers  (2023-07-06T17:58:40Z)
• Representation Consolidation for Training Expert Students [54.90754502493968]
  We show that a multi-head, multi-task distillation method is sufficient to consolidate representations from task-specific teacher(s) and improve downstream performance. Our method can also combine the representational knowledge of multiple teachers trained on one or multiple domains into a single model.
  arXiv  Detail & Related papers  (2021-07-16T17:58:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.